Yes, enemies will destroy farmed plants and saplings. Not, however, on purpose but as collateral damage in trying to get to you. Saplings and seeds in the wild are not targeted.

Some structures, like rafts, boats, workbenches, and portals, may be deliberately attacked by enemies if a player is not on their radar. The rest is collateral damage.