Two reasons:

Mods. Do you have them running? Do they have them running? Take them out - both of you. (Verifying files does not remove them.) You either have to have the exact same mods and settings going or nothing at all.

Crossplay. Are either of you playing console? (Is steam deck console?) You should be able to play together if you've got the crossplay option ticked on your server.