Tbh for me personally it fails on the two important parts:
1) It's not immersive because physics break everything
2) it's boring

I also disagree with that it's hard or annoying, you literally just need some stamina.

I guess the main source of contempt is, for a completely optional activity it's just not interesting or polished enough so there's little incentive to do it, unless you really want that fish food for whatever reason.