Not broken.

More like TOTALLY AWESOME.

Find that little patch and build a spike wall, and a trench/moat and or earthen wall around it. Then laugh at them. You've got a captured Troll and grey dwarfs in the center of the relatively safe meadows! Consider the glass HALF FULL (not half empty).

Build cool tree house that looks down into the enclosure and start charging admission.

There are solutions to nearly everything in Valheim.