I've done some more tests. It looks a lot like the client is sending world data to the server, not just player input. Even without other players around a person with a limited upload capacity (e.g. DSL) can experience lag on a dedicated server. The area you're in will be in sync, you can open chests etc but as you move away you'll see dwarves, deer or items you can't interact with. After some time they'll respond.

This implies the client is sending world info to the server, once the server vettes the clients assertions the server sends it back and you can interact with them. If the bandwidth limit is severe enough as you walk on you'll eventually come to an empty world, no trees, no rocks, no animals no nothing just rolling hills and grass. As you keep moving your client can't send info to the server fast enough to keep up and just gets further and further behind.