Just because you can, doesn't necessarily mean you should, though. It's a lot less effective to mine those deeper veins. (More time consuming). And it's going to end up like a huge scar in the landscape if you don't use some of that stone to fix it.

But... that doesn't mean you shouldn't, either, just means it might be worth considering.