Most mobs in the game lack capacity for tactical thinking or reasoning, and those which do not (like fulings) probably treat you, the viking, as invaders. So you're already notorious, but not notorious enough to warrant those few creatures with capacity for social construction to abandon the defense of their homes.