It can be tricky to use the ladder on any boat. Thw target area isn't as big as one might think. Also, you are bobbing in the water and the raft is bobbing in the water. Keep calm. Thrashing around uses up stamina and hastens drowning.

Another possibility: Rafts are small; did you friend get on first and then stay by the ladder? Did he move to the mast or tiller to make room for you? I have only played solo; I can't give you a definitive answer. I do know that, in those abandoned stone towers, a skeleton can stop at the top of a ladder and keep you from reaching his level.