Also they do hang around the caves (not always), unless you have kill the Troll inside. Than they stop showing up in the area around it.