I have the 'center'; I did as you said and built off a exact initial marker stone.
Its just not symmetrical. The symmetry error goes all the way down, unfortunately. I think I got thrown off with the terrain raised support spikes which were hard to place precisely and I had to adjust a couple of times. I used that a bit with the entry-ways -- those are on the 'extra fat' walls.
Do you think maybe I should loft like a tall pole off the center and then try to come down off that rather than trying to build up off the walls...?