The whole being able to use the ladder while swimming in my experience is lacklustre at best, you have to be in the EXACT right position or it won't bring up the prompt. I do hope they extend the range of which it can be used.

I recommend avoiding the raft all together personally, especially for living islands and long voyages as if a serpent attacks you in a raft, you're done 100%. Where if you were in a Karve or later the Longboat. You have a chance of killing them before they destroy your boat.