Secure your base with a ditch. Little wood fences don't work very well past the meadows event. It will take some time so just build a ditch all the way around your base with room for some gardening within the area. Dig deep enough that you get straight sides that critters can't just run up and down at will. Then go kill the elder. Dig down under his alter area and put your portal there and can even set up a sleep area with bed. Some folks even make real bases under the alters.

To kill it, use your trusty axe, it's a tree, when you can and play hide and seek behind pillars rest of the time and give him some fire arrows. He doesn't turn fast so go in back of it. If it gets too intense, jump down in your hidey hole and regen. Move away from the womping willow branches but don't use up all your stam. You will be fine.

For all those that say it's easy, good on them. We are all different in our playstyles. Ignore them, it's just humble bragging anyway. Kids.