I had this problem in my big plains castle and what I eventually figured out was happening was that when I was outside my base but still nearby the game was partially unloading my base and causing bits of it to become unsupported.

The solution was install a wood-iron frame that anchored all the stone in two places. Like a single frame piece might look like this

S = Stone
W = Wood-iron
G = Ground

So no matter which way it's facing relative to me there's always one side anchored to the ground. I never had any problems after that.

Now I always think "How is this going to be supported if it's only half-loaded" and I haven't had any problems since either.