I modified our server transfer limits and most of the lag went away:

https://jamesachambers.com/revisiting-fixing-valheim-lag-modifying-send-receive-limits/