I just kill the annoying graydwarves and get on with my day. I had a mountain keep that would occasionally get drakes, but it wasn't too hard to shoot them down. Never built in the swamps, so I don't know about Draugr outside of the Bonemass raids. Probably the hardest thing was the fulings and deathsquitos, though they'd always come in from the distance and never spawned in the base, not like the graydwarves. You know, those ULTRA RARE (according to the rune stone) monsters that throw rocks and generally are more a nuisance than anything else?

Honestly, I never see this issue. I see them in my yard sometimes, but never actually seen one just pop in or be in my house. I'm a grumpy old man type, so I always come after them with a spear screaming "GET OFF MUH LAWN!!"