While I haven't had the issues you're talking about with building, especially in the start of the game. I'm not sure what mobs are strong enough to destroy your buildings? I have been attacked by huge waves and just chilled behind my walls. They never make it inside.

Now, I do use a trainer. I would recommend "WeMod" it's free. You just turn off and on different features. For me I use unlimited weight because if you think you're aggravated now, wait until you start mining metals. Dragging a cart around to haul small amounts of iron is extremely aggravating. If not for using a trainer I would have 100% quit at the bronze age.