To answer the question directly; no. It is not worth relocating a base. As others have said, you should make a few during the course of the game.

I have a main base with everything I needed to get through the bronze age; it has a nice portal hub and a full indoor smelting/forge operation. It is in the meadows, fully moated and impervious to attack. Right on the water with a nice boathouse.

When I hit iron, I built another base similar to my first but scaled down to just include living essentials and a second smelting operation specifically for iron. It is located in meadows just outside of a swamp with many crypts.

I found no need to build a full base in the Mountains, but I did renovate a nice castle for my time there. I placed a dock and small camp across a meadow at the base of the mountain to load my silver ore and boat it back to my main base since it ended up not being all that far.

I am now in the plains and, again, see no need for a base here, yet. Black metal was, again, easily loaded into a boat and shipped to my second smelting location near the swamps.

So don't overbuild early in the game except for your main base. It will serve you well over time. With full defenses you can portal back there every night to sleep and rarely ever get raided. If you do, just relax and do some cooking until they go away. I use it constantly.