Sounds like you need to learn how to make proper defenses. Also, place workbenches in your walls or wherever else you can put them out of the way.