Yeah once you summon a boss it stays spawned until you kill it. A friend and I went to fight one of the earlier bosses in our first playthrough and accidentally spawned 3 bosses. That was fun. Suffice it to say, we're very careful now when summoning bosses lol.