Happens to me but only on dedicated servers, it doesn't happen when I play single player unless there are items/build pieces loading in.