If you don't need whatever bonuses, armor vs no armor is the exact same I'd you don't get hit. If you can pull it off, go for it