Trolls are wrecking balls if left unchecked and I'm pretty sure none of the structures we can place to date will allow you to safely afk a raid without losing anything.

That being said, wood buildings fall like a playing card house; at least the stone variations can survive a little bit.

Is it always worth tearing down and rebuilding with stone? No, at least not for material and time efficiency for a protection standpoint. Rarely will it be that useful.
You'll be better off making a outer stone wall for protection for another buffer layer between them and your building.

The best use of stone is building vertical or just the aesthetic.
The mistlands building pieces seem to be immune to a few more elements and have a bit more vertical stability, but are listed at the same durability as regular stone.

Kind of wish the mistland pieces looked a bit more like the regular stone, if using them together in a build the color and texture differences are too different and it stands out dramatically.