I followed a way explained in the BepInEx manual to install the plugins outside of the Valheim directory.
I created a /unity/BepInEx_Valheim/ directory in what should be \AppData on Windows (/.local/share/ on Linux), and put BepInEx and all its mods and libraries inside. Then there's only a path to change in a config file.