You can certainly game with modded and unmodded players. Ive done it, maybe not with all mods.

Some mods are server side , most are player side.

*edit : my experience is on a privately owned dedicated server.