yes, it matters. They all kill stuff, but consider a few of the many things...
- the toxic axe has poison dot but that does not stack so your 5 hit combo leaves it ticking for the same as a drive by 1 hit. The poison damage is effectively 0 after 1 hit.
- a lot of stuff resists pierce, which is because that is what bows use and cheesing with a bow is frowned upon a little bit.
- elemental and spirit damage are pretty handy. spirit for undead, which are plentiful including at least 1 boss. But elements and spirit can also be resisted or near zero.
-frost slows enemy, making them much easier to fight
-frost does extra damage to wet things.
- blunt is slightly > slash for getting the job done due to vulnerable/resist pairings. This is very notable in the swamp, and less outside it.
- aoe weapons have a number of exploits and strategic uses from finding things to stunning a group to fighting through a wall.
- knives and some other weapons have a high sneak attack modifier for a different style..

and so on. its not that complex -- with a few exceptions** anything you pick up will get the job done -- but there are tradeoffs in effectiveness, style/approach, and whatever makes you prefer one over another.

** in general anything that is a tool is bad, like trying to fight with a picaxe, and to a lesser but important degree, axes are generally more tool and less weapon here. Harpoon is a tool. The butcher knife or whatever is called for killing your pets is a tool, as is the fishing rod and the beer mugs. If it doesn't seem to kill things, that may not be its primary purpose. Axes are a grey area here, esp early game, but I stand by lumping them in the tools.

you can't go too far wrong using {sword, mace, ateigr, spear}. Those all are popular for various reasons.