Just to put it out there, nothing under "building" has any benefit per se. You could literally finish the game with just a few wood panels and roof pieces for your bed and workstation. It's all about building "pretty " things if you're into that.

But yeah indisputably stones are stronger than wood - and black marble is stronger even. Nothing will completely stop ennemies so you can't just rely on your walls to fence raids.