I've seen them growing on the stone paving ground as well, but never tried to pick it up, to see if it comes back.
Well, check real life, that stuff is growing everywhere, even coming through asphalt roads...
I wonder about a similar thing, I always have branches spawning in my house, even though there are benches nearby. Any way to avoid that?