They won't one-shot you if you're not trying to fight them in low-quality armour, and even troll armour is good enough if you use parry. Just never go at night.