There's no build limits that I can think of that would result in random destruction.

It's possible that you reached the limits of the structure that you built and it just took time to cave in. If you have items like chests or benches on those parts of the structure that fall apart, they will fall apart too.

The only other thing I can think of is that you missed a raid event of some description, but if that had happened, it stays active until you return to the place it started so you would have known.