That never happened to me. The only time an enemy got into my base after walling was when I forgot to close the gate.
I didn't delete the workbenches after building the walls (so that I could repair them anytime), so that might be the reason.