Enemies may not destroy saplings, but you waving a mace around will destroy your sapling. Its unfortunate that this happens, this is why fences are so great!