Again not been my experience. I get you that its a bit unnerving to see "Its not taking the bait" when you can see the type of fish you are baited for right in front of you, amongst others.

But that Pike or Tuna or whatever is: NOT taking the bait. So wait and the "right" fish will take a nibble and with a click you can start hauling it in.

You are right there is little reason to fish outside one area until there is more reason to do so, IE recipes. The hat is insufficient. The angler fish recipe is a great start but disproportionately hard to get for its effect.