Greylings and Greydwarfs are practice mobs. They are for skilling-up with your weapons. To that end, they need to be plentiful. Being annoyed with them is like being annoyed with targets.

I can swap my ax for my bow and shoot a greyling with scarcely a break. Or I can just include the little fish dropping in my swing.

Being annoyed is a choice.