because a flag or a pole is different from a workbench or campfire or whatever?
there are lots of ways to deal with them. Loft up, moat down, island, anti-spawn, wolves, ignore them, more.

Yes, they need a tweak, game isn't done yet. My newest project is near ashlands in BF next to swamp and its nonstop enemy. Last time I went to chop wood for the roof, I got, literally 4 to 1 pieces of wood from dwarves vs trees.