Momentum is a thing. To stop the boat without jumping out, you have to cancel out the momentum. Cut the sails as you're going towards shore, then stop with the rowing, and when you get on the last stretch, hit reverse until the boat stops, then immediately kill reverse.

When you get practiced enough at it, you can stop the boat EXACTLY where you want it. But if you're not 100% stopped, it will continue moving unless you jump out. And if you haven't jumped out, it *can* drift if something happens.