Blocking skill will actually be a huge help when trying to cross the parry/block thresholds for progressively higher damaging enemies. And if you don't have the stats to execute a parry then you are missing out on a tool that is great for both dps and crowd control.

Probably swimming is the most useless skill. You usually want to avoid swimming even if just to not get wet. You are highly unlikely to be forced into a situation where having a high swim skill can actually save you. Some mobs don't follow into the water but you never have to swim far and you can preserve stamina while you wait by simply not moving. And although I've never needed to do it myself, I'm pretty sure you can circumvent the stamina problem by exiting the world and re-entering with full stamina.

Most of the rest I think depends on what you use and don't use in combat. Like I don't use many knives, spears or polearms. Often I am too impatient for melee sneak attacks and am happy getting my pierce damage using bows.

And of course unarmed is crap too. It has the same percentage damage scaling as all of the weapon skills, except it never scales off anything better than your bare fists. No gauntlet or knuckle-duster type weapons. The difference between a 0 skill and max skill punch is like 3 damage. Just like with swimming it is a skill that you should actively avoid needing to use. There are times where you want to hit something without wasting weapon durability I guess.