Vortex will not be able to manage mods installed manually. As far as Vortex would be concerned those are original game files. Not to say that you could not start using Vortex as is, but I'd suggest uninstalling, deleting the game folder, and reinstalling from scratch. If you choose to do this do not forget to delete the Valheim folder as uninstalling through Steam will leave all the modified, and extra files from your manually installed mods. The main benefit in doing a fresh install is being able to drop in and out mods, and tracking down conflicts. There are a few mods that still need to be manually installed, but you will see that on their Nexus page. I use Vortex myself, and currently have about 80 mods installed. LOL, Only real problem I have is having to remap and remember so many different hot keys.

There is no real problem with doing all this manually, and not using a mod manager like Vortex. It just gets increasingly difficult as you add more mods, to change what you have installed. Vortex automates all this.

***Edit*** I forgot to mention, one of the best aspects of using Vortex is you can have it check for mod updates, and have it run the update for you. This benefit is twofold, 1st, Game modders sometimes add features, 2nd, but more importantly they also squash bugs, and update when the game is updated and breaks their mod. You have to remember, this game is still in beta, and as such is getting new revisions often.

If you have any specific questions, do not hesitate to ask.

Good luck from a fellow older gamer.
Joe