The game is inspired heavily by Viking mythos and Vikings did not live in massive castles they lived in small 1 room farm houses and communal long houses. You didn't start seeing larger constructions until after Christianity started to spread through Scandinavia.
https://www.followthevikings.com/discover/daily-life/houses