Aha ok. I have seen the same thing with the forest i planted around my mansion when I coming home from the sea but it never really annoyed me that much.
From the distance I still see the naked hill how it was in it's original state, then the trees i planted starting to popping up everywhere.

The only thing that annoys me is that the bonfire I want to have as a lighthouse, rendering the same time as the rest of the mansion. So it's pretty much useless as a lighthouse.

Edit: As for the loading time it's never a problem for me as i have a separate 1 TB M.2 disc for gaming installed.