Depends on your playstyle. Here are my three main bases:

1- newbie base near your initial spawn
2- main mid-game base. Running distance to swamps to get iron.
3 - late game base in the plains.
+ several mini-bases near each boss and the trader consisting of a protected workbench and portal.

Out of these the most important one for me was the second one. It is 100% worth it to relocate to a location with easy access to iron IMO and makes your whole mid-game pahse much easier. As for silver, it's up to you but I don't think it's as important and it's less advantageous to relocate just for this.