I stopped worrying when I got smacked by a log wielding troll after getting hung on something.

Pretty sure I lost 5 or so levels across the board so if it's going to punish that hard I won't bother.

It's not too big a deal, the difference between 1-100 in terms of weapon damage is 3 times your starting value. Not really worth grinding the skills specifically for that.