Originally posted by Grand Forks:

what is the speed of your internet? Noticed when building excessive building with multiple players I may get dropped but lag is horrible, Most of the time that happens its low connection speed/data rate from the host.

This is another thing as well.
The host for any particular area can change depending on who was there first. So if you go to an area where someone with a weak internet/computer initiated that "zone" you could have issues as well.