My take on this would be to add a roof abover or underneath the platform where you stand on, so if at all, only the top floor will get damaged. There are other ways tho, like Placing the X-pieces into the walls I think. Or you could use the stake walls, they won't take damage from rain and have more health.
The version without a roof might be better for some people since roofing can easily block your sight, at least that's my experience.