When you do any boss start first thing in the morning so you are not fighting at night when many harder enemies can spawn and join the fight.

Before you start the fight, scout around the area and look for and take out troll or other skeletons and greylings that may hear the noise and come running. you can place a few workbenches around to stop any enemies from spawning in, but be careful - if you have 3 or more base items in an area where you are (like 3 workbenches, or workbench, portal, fire pit) you can trigger a raid, The Forest is Moving , where you will have way too many things to fight in a open area around the Elder alter.