Not sure how much you lose, i think its "only" 1 point of each skill per death. Atm difficulty seems to be fine but i do see a problem with stuff that you rarely use, like swimming, will just never be viable when you lose points in them from time to time.