May be not a full castle but yeah I usually have roofed pens for that reason!
of course soon as I sated doing that I had an ooze stand next to the pen and spew poisons on them last raid so... yeah I feel your pain