They do chase quite a bit. Even if they are 0 threat, it's annoying to hear rocks being thrown at you as 3 greydwarfs chase you for 3 miles straight.