Slight penalty to consider: You lose rested buff, same as when you log out / log in for a single world.