If you plan to make a lot of the stone walls and such later in-game, you'll burn thru stone fast depending how big youre building.

As for the stone piles, I've always received the 50 back from it, same with the wood