It seems likely that you've just gotten unlucky with the number of spawns around you, and/or you're mining at night.

Mining and chopping trees has a large "noise" radius, so that's another thing you might be falling afoul of. See https://valheim.fandom.com/wiki/Creature_senses#Noise

Make sure to sleep at night to minimize the night-time "wandering monster" spawns, because that's a thing. If it's daytime, then you should only have to deal with an influx of creatures early in the day; once you've attracted all the local baddies, they should stop showing up.