The "low graphics" (pixelated textures) are by design, not for performance. Build pieces for example render all sides, even invisible ones. Lighting, smoke, etc effects are all modern and graphics churners. There's also a lot going on in the background: calculating angles for effects, spawning mobs/trees/dungeons/etc in the distance, etc.

In short this is a demanding game. However the RX 580 is listed as recommended on Steam, so you have every right to expect good performance. IMO Steam/Iron Gate owe you decent performance, or a refund (and retention of the game) as well as updating the listed system specs.