I built too close to it initially... now it's too much work to move. So I walled it in. Leave enough room to build workbenches / fires just inside your wall to keep enemies from spawning within the center. You may still spawn a random deer or skeleton occasionally, but at least I haven't had any trolls there for a long time.

If you still have the freedom to move your base, I suggest not building too close to the center.