Bosses have a red circle on the map once summoned--that's pretty much their aggro zone. So if your base is in that circle, or very close to it, getting attacked by them there isn't a surprise.

Easy way to stomp the elder next time. Prior to summoning, drop lots of campfires on the outer stone ring of the facility and the ground just beyond. You should have lots of rock from all that mining you did, use it. :D Then, you simply set up opposite side from the elder, shielding yourself from his attacks and sniping with a bow. That annoying roots attack he does is completely nullified by the presence of the campfires, and he generally won't cross them either, so you can snipe with impunity (just dodge his ranged attack behind the indestructible pillars).