Rafts are dangerous contraptions. Ill use one early if my island is too small, to go from a shore I am on to a shore I can see. Otherwise I wait on the karve which is my go-to ship, the longship is too big for my tastes unless hauling a lot of ore.

The sea is a bit dangerous. Storms and waves won't hurt you but the serpents and stray biome critters will.
Honestly, the serpents always swim off if I shoot them 3-4 times. They are 'mostly harmless' unless in the raft.
What gets you out at sea, more often than not, is simply the stray biome critters.
the swamp's leeches will swim way out and chew on your boat. Undead archers will go under and shoot you from the depths. Trolls can wade out a fair distance and smash at you. Small humanoids and animals will swim at you but usually cannot catch you or do anything. Dethsquitos will dive bomb you and they can sink a boat really fast, you need to shoot them down if you are unlucky enough to get one to chase you.

All that boils down to your original thought -- storms, night/dark, fog etc are dangerous just because you cannot see the shore type ahead of you and may stimulate some nasties into sinking your boat. Its best to explore on clear days when the sun is up. If you are caught in a storm, you can stop where you are if you are lost or confused. The minimap is your friend here, you can 'see' what you can't see using it but the range of discovery is a bit limited.

Enjoy the sea, and be very wary of boating into the unknown, and you will be just fine otherwise. If you didn't know, 1 click forward is row against the wind, one click back is reverse and inverts your turn controls like a car.