The server only progresses so long as a person is logged in.
As far as i am aware the "persistent world" mods for servers aren't worth even bricking your game over because so far, that's all they seem to do.

In regards to difficulty, it actually comes from how many bosses have been defeated on the server, not how many people are on... among some other details (like serpent spawn etc)
Keep in mind too, each boss defeated creates a new wave from the future boss that not only you but new people will have to defend against.
This means, you could have a fresh spawn on your server that gets attacked by the "swamp" in meadows because you and your friends have defeated the Angry Angry Ent.