I'm renting a server atm and playing with 4-5 people, we have a village medium area 5 buildings and a town large area 7 building and a great hall as well as a number of outposts and camps and we get the odd lag spike with trouble opening chests and some damage to ships when sailing but for the most part no issues and the lowest my fps have gone is 40 fps using a 1070. My game setting are all set to medium.
It seems alot better than when it first came out and as long as we don't raise terrain too much it seems to be pretty smooth.