Crystal were a big deal pre-cave, so in reality golems sort of got a nerf due to the cave addition. Before that, if you wanted a window, you had to deal with them.

But not worth the effort is correct. golems are excellent at killing wolves and distracting drakes while you mine nearby safely out of their reach, but actually fighting them is a huge waste of time.

Gjall are not worth the time either, and their claim to fame is clearing out towers so you can take the npc item.

I would not complain if they were buffed to drop something useful, but is it 'bad' that some monsters are best to avoid rather than engage? Is that 'playing smarter' ?