Sometimes they chase an enemy and then get stuck down a hill or something but they're still attached to the ward so even if you clear the camp, you may have missed that one somewhere else and he's still angry with you.