You can stop the drake attacks by defeating Moder, but the bat attacks are permanent once you enter a cave. I've just been avoiding caves, which solves that issue.