The mist itself seems to be the source of a lot of it. When I've modded the mist away with the MistBeGone mod, the FPS dramatically improved. Don't know if that's a solution you want to use, though.