In my experience I have only found silver where there are Golems, obsidian & wyrms (drakes), and silver always seems to be at the level of obsidian or higher. Nearly everytime I find silver there is a golem spawn or two nearby. Golems can be a bit of a hassle as they tend to respawn rather quickly, but if you push them into a pit and leave them there, they will not bother you.