Build a mole or jetty by raising the ground out a few meters in a straight line, then extend your wall or palisade on top of that. Strong swimmers may get around it but it will take them some time. You can snipe them while they make the attempt.

I do note that if you build stone walls to the waters edge, those ground shaking trolls prefer to wade around the end of the wall rather than break through it.