This is why the Sneak skill is so useful...
Also, ditch health foods for as long as you can, keep drinking that Honey. I swear 75% of the time people get hit, it's because they ran out of Stamina.
Stagbreaker also melts them if there's no Brutes present.