I had a walled off base with crops, and somehow greydwarves kept getting inside and wrecking my crops anyhow. It seemed to happen if I was near the base at night, but not inside and far enough not to see it happen. Occasionally I'd find a dwarf inside the walls, but I could never figure out if they were spawning there, or getting through the wall somehow.