install the mods on the server without r2modman?
Start the server as usual with start_headless_server.bat?

I'm not using start_headless_server.bat but NSSM to start my server directly as a windows service and the one mod required on the server works fine (noAFKevents) as long as it and BepInEx are "installed"(=extraced) into the correct folders each.

Oh and (I) don't use "desktop Steam" for a dedicated server.
Just use SteamCMD(.exe) like for most other dedicated Steam servers to download, install and maintain an account-less copy of VH's dedicated server (see official Wiki for examples etc.).