Those stone towers are usually in bad shape and you can't repair them until the next age up. I put stake walls and a trench around mine as an added precaution. The trench will keep trolls out, but they still have a long reach. A log wielding troll was still able to wreck my carrot garden from across a trench. Leave more room than you think you need.