Animals and Greydwarves are afraid of fire. You can gain peace and quiet by putting down a few fires or torches. They'll still throw stones at you from a distance. Shamans and brutes are not afraid.