A lot of it will depend on hardware. On the server I play on, there are 8-10 of us and we formed a single village with each of our homes, as well as a workshop area, boathouse,metalwork. portal nexus, and metalworks. Probably 12-15 medium to large buildings, with lit paths connecting them. My FPS drops from 100+ to 50-60, but I'm playing with max settings on a decent rig. Others have it worse, dropping into 30fps or lower but it's still playable.

Short answer, yes there will be some lag (fps drop is a better term) but it's much better now than it was a few months ago!