The skill lose doesn't matter much unless you're dying a lot and so it builds up. Even if it was a big hit - and it's not - dying infrequently wouldn't be a huge problem.

I guess what I'm saying is try not dying.