I'm going to guess that you have made major modifications to the terrain with the pick and/or hoe to make a large flat spot to build in.

In the current state of the game, terraforming eats frames when you do it in large chunks. The effect is localized, only affecting you when you're near the terraformed area, but the effect can be quite pronounced, especially once you've built a significantly large structure in the area. The core of the issue appears to be how modified terrain is stored in memory (and on disk).

The community is hopeful that it will be resolved in a future patch, but the workaround for now is to avoid raising/lowering terrain in areas where you want to build.

On the bright side (and as you have already noticed), if you move away then you get your FPS back... you can just make a new base elsewhere in the same world and avoid the "broken" area to avoid the frame drops.