I did a playthrough with just troll armour and knife - assassin style. Zero speed penalty. Almost zero stamina. So fun.