The greylings and greydwarves (except brute and shaman) will avoid coming close to fires.. the distance is, I believe, equivalent to the fire's warming radius (unsure with standing torches what radius is).

But yeah, the biggest problem is having a harbor that opens across from a spawn point. If you absolutely had to have a harbor like that, it might be worthwhile putting some spawn prevention on the opposite shore.

Fires, and for that matter earthen walls/breakwaters, will only help if they are in the mob's shortest path. It gets even stranger with skeletons as I believe they can spawn in shallow water.