Stone walls (or any type of wall really) are not meant to stop monsters from getting to you at all times. They are there to unlock new build pieces, new architecture, and to buy you some time to get out of your base and face the invader(s). Stone is almost four times more durable than wood. Stone structures, artfully incorporating wooden build pieces, look awesome. :-)

If trolls are your bane, they do about half as much damage to stone than they do to wood. And since stone offers superior durability, a reasonably fortified base will ultimately provide vastly superior protection.

Also, this is Valheim's official knowledge base, endorsed by the devs themselves, and an awesome resource on the Valheim universe -- http://valheim.fandom.com/wiki/Valheim_Wiki

...much of what I said here I took from the wiki.