non series X version of xbox seem to lag fps wise a lot from what my friends tell me, in terms of server lag you could try following this guide for your own dedicated server to improve the delays https://jamesachambers.com/revisiting-fixing-valheim-lag-modifying-send-receive-limits/