They are one-hits and don't do any real dmg themselves. IMHO nothing to worry about, just a slight annoyance when you are at 0 stam and not rested while chopping down a few hundred trees to root out a whole Black Forest area... other than that just shrug them off or let them hit you and continue whatever you are doing - eventually they will run into your line of fire anyways and die while you chop away.

Personally I find it more annoying to have to clean my inventory when accidently picking up their droppings.