Armor strength is very strongly tuned, so going lighter is almost always a bad idea. Matching half the enemy's attack power will double your effective HP, and matching the enemy's full attack power will quadruple HP. Bonemass power(and resists) also apply before armor reduction, allowing the armor curve to double dip.

Fenris pants have an acceptable trade of losing some armor for dropping the -5% and gaining +3% speed. Adding a Fenris chest is risky but doable. If you're cave diving, or trapped in battle, stick to max armor at all times. For exploration, a little speed definitely helps.