lol the dvergers are neutral; you can be ripped apart right in front of them and they
don't do squat until they are attacked or course if you hit them by accident they
will kill you

It's possible that what happened is the Gjalls and rogues had a confrontation and
your farm was 'collateral damage'. But still I think you need to be 'in the area'
for that to happen.

I think the trick with any farm is plant it and leave. Don't hang around the area;
come back in a few days and harvest stuff. Repeat. Attacks don't happen
if you're not there or none of us would have intact bases.