There is more than one altar for every boss on a world although only one will be marked when you use the first knowledge stone. So if it looks really ugly where the altar is now just go explore more swamps and you will ultimately find another altar in an area you might prefer to fight in.