From what I've heard, it should only depend on what bosses have been killed. You could even bring the bosses' heads and hang them on the altars and it wouldn't count if they hadn't been slain in that world.