It's possible that there was a delay on the spawners. The Draugr all seem to start in the buildings with the spawners, so if you didn't get rid of those, then that is why it was "invaded" when you went back.