Stone walls have 50% more durability than wooden or stake walls. Where you gain some ground is that you can build with the smallest stone blocks and the enemies have to break through multiple of them (instead of just one) to get inside.