I made a ditch around my base about half the height of a troll, keeps them out no problem. I take no damage if i walk off into it, but if I jump and land in it, i take maybe 12 damage...Maybe you are making your ditches/moats too deep?