You need to be able to deal with swamp before killing the Elder will do you any good anyway.
Otherwise all the Elder gets you is nastier raids.

So I would recommend getting started sooner and crossing that northern swamp. Just mark where you need to go back to post elder (and maybe leave a portal somewhere handy)