I've recently gotten some scales and a trophy by delivering finishing blow to the worm by jumping on its back for the attack while still on open seas. What the optimal weapons for that are, I'm not so sure, but Frostner or Mistwalker deals the most damage thanks to the serpent's frost weakness. Jumping at its face to deliver a killing blow should ensure all or at least majority of the sinking loot dropping in your inventory instead of seabed.

This all obviously requires pretty good awareness and understanding so you won't let the serpent destroy your ship while whittling down its health and waiting for it to get close, forgetting to bring the sail up and becoming stranded in the sea (at the mercy of the serpent or otherwise), running out of stamina and drowning, and other factors.

https://steamcommunity.com/sharedfiles/filedetails/?id=2937610470

The shallows method described above by Hobo M. is probably the most reliable method overall. If you try dragging the serpent to shore and you happen to run out of stamina at just the wrong moment, you may never catch the bugger again. But in shallows a healthy serpent may yet try to fight back and not escape.

If you use Moder's power, you may be able to make a sharp turn at a beachhead so you can pull the serpent far enough to shore so it won't have even hope of escaping. So once again, it boils down to terrain... or absence of it, in this case.

Frost damage also has the advantage of not harming the ship to such extent as some other damage types, as well as slowing the snake down. Longship is also resistant to piercing weapons, so you could use a spear or atgeir as well if boat damage is your concern (they are not resistant to lightning though).