If you plan to build structures with stone later in the game, you may want to keep that stone (just build rock piles somewhere). It takes a LOT of stone to construct buildings, so it could save you some time down the road.