Then is normal , raids can be possible even with a little cabin
And there are is abandon ruins in mountain that are defend by skelletons and swamp people too
Im meadows are abandon villages with a lot of swamp people spawners so be careful
Swamp and troll raids can happen in every part of the map when you already killed the 2nd boss and a troll , in a place that have at least 3 buildings ( workbench , bed and another thing , but can be much more thing too , not just that )