Are you staying rested? Being rested gives you a major regeneration bonus for stamina. In Valheim, you have to stay rested at all times, or you're going to have a bad day.