Are they really random pieces or are they the same ones every time?

There is a rare bug that causes this, that I've only seen once in my various playthroughs. I had one stake wall shack on the edge between a black forest and a swamp, and one specific wall would always be broken when I returned to it. The rest of the building was completely fine. That one piece was supported properly, but no matter how many times I put it back, it always broke.

That border explanation above sounds about right.