The sea is always right! It's your fault you wanna sail in the wrong direction:)
But seriously, sometimes it kinda feels that way. You sail around an island and the wind always changes in the worst way imaginable. Just bad luck. It's RNG. Many times the wind changed in the right direction in the moment i started my journey.
Kill Modr, sacrifice the trophy and you will get a buff to change the wind in your favor. Until then: Good luck! :)