If there's a Gjall in the area they can destroy your mushrooms but I think you also
need to be in the area. I put my mushroom farm next to one of the magic roots
so I have sap also. A small shed with a workbench and 2 chests for storage built
into the root and double high stakewalls around about 10 mushrooms.
The portal is a short distance away in a piece of Black Forest. Just a box wood
house with a fireplace and lots of storage. I hid it under Firs because there used
to be a resident Gjall apparently patched out. Had a nice little private war
going on for awhile.

Gotta pick a good spot. Are you sure it's all Mistlands? You can plant the mushrooms
in the Black Forest (or could) but they just die. The minimap has to say Mistlands.
Plant just one to start and see what happens. It doesn't appear that the mushrooms
respawn in the wild so you have to be careful with what you have.

The Seekers don't wander around too much thankfully but if you can hear their
hissing; it's not a good place to set up a farm.