build a 2 block high stone wall around your stone building so they have chew through
that first; double thick walls work too

this will work against everything but blobs (which jump) and of course trolls

^! I keep forgetting about the big hammers but be aware that their area effect
will damage your structure also.