Yeah invasions can happen at any time. Pause if in single player or log out if on a server the next time you need to step away from the game.