I think it's more of a CPU load thing, since the game secretly runs Borderless Windowed even if you select fullscreen. Whether there's momentary slowdown or not seems pretty random in Valheim (not strictly related to what's happening on-screen) and that also points towards a CPU thing rather than GPU or anything else.

But yes it still happens, from time to time. It's not as noticeable to people with mid-range or better computers, ofc.