I think your #s assume you get 1 fish for every bait. That is NOT my experience in general.
You can't get enough of these guys to justify the time sink unless you enjoy fishing. In the time you can catch 1 stack of fish and bread, you could have planted and harvested the stuff to make (assuming you did something else while it grows) 20 or more stacks of salad.

I do recommend getting enough to make some for boss fights or even troublesome areas like the mines, but its just not worth it to me (and I do like to fish) to catch enough to eat this all the time.