According to the wiki, fish check for bait every 5-15 seconds, and that check has a 10% chance to make them go for the bait.

So in other words, another strange design choice that isn't respectful of a player's time.