Through similar experience I have learned to always leave an icon where I moor my vessel when exploring. Valheim Plus has a feature that places boat & cart icons on one's map wherever they are located. That feature and the one that allows extended zoom out while sailing are 2 of the QoL features I use V+ for.