I always keep the most recent, heaviest, most-upgraded armor on me and I wear it when I go into areas with strong enemies or whatever section of the game I'm currently up to. When I'm running around, gathering, building, exploring, I only wear the helmet and cape, which impart some benefits but don't slow you down. I just quickly equip/unequip the chest/leggings whenever I need the extra defense.