I play off a USB passport drive and it only hangs for maybe a full second. I've not really noticed a difference between many different worlds. Fully explored worlds maybe half a second longer?

10 - 20 seconds seems a bit excessive, if you are sure about that time frame then maybe you should verify local files. Library - game - properties - local files - verify

Do you run a lot of mods? I run the game unmodded.