They won't come back if you've destroyed all the spawners. I find the camps to be ideal places to build your own housing since many of them have large patches of relatively level ground and trashing the existing buildings gives you a pile of material to build with. Look at the buildings carefully before you tear them down though; some may be worth salvaging and they're a great place to pick up ideas on what and how you can build.