Moats are your friend, followed by earthen walls on the inside of your aforementioned ditch about the time you get to the swamp. Unlike built walls of stone or wood, they are impermeable - and garden plants take no damage from troll thumping, so ring your base with your garden. atm walls & moats stop all but the most enthusiastic blob - but they don't do much and die easily. And of course workbenches. Best way to test your coverage is to go into build mode and wander around every inch of your base to see if there is complete coverage. Even a skinny little area will spawn a big fat troll. Who knew... And yeah, pause is the way to go now that we finally have it. Murphy's law is alive and well in Valheim.
Depending on your level & gear, just use your best axe and chop the abominations down. You can also take them over to a sertling spawn and let them toast themselves too.