It'll definitely affect you as you add more weight. Using the Eikthyr buff can help a bit with stamina, but the weight increase will still be noticeable as you struggle up hills.