I was modding with Vortex too, even if you uninstall the mods with that it still doesn't really get rid of anything, so avoid using that and use a different mod manager from Thunderstore.io like r2modman. Or just do it manually if you're only using a few simple mods. I didn't really need a mod manager for the few basic ones i wanted. It's really simple to install mods manually.

Also I found when you uninstall the game, it doesn't get rid of any extra files you put into the directory. I had to manually delete everything from my Valheim folder so it was completely empty, then reinstall.