You do NOT need multiple installs. Use R2Modman. It allows you to set up an infinite number of profiles for a single game. You just load up the mods you want for each profile. So if you want to play vanilla, just load vanillia. If you want to play with all building/creative mods you can create a profile for that, etc.

I have several profiles for Valheim as I play on my own specific worlds for creative building with all creative mode mods, worlds I share with others for adventuring with stuff like Epic Loot, etc, and dedicated servers that all have their own mod packs too.

The only thing you have to be careful of is to make sure you don't load up a world seed with the wrong profile. :)