If you get the message that the fish aren't taking the bait that means that particular fish takes one of the biome specific baits. The regular fishing bait you buy from Haldor only works for Meadows fish. You have to craft the other types of bait and the recipes for the next biome's fish bait should unlock when you catch a specific type of fish in the current biome. For example catching a perch should unlock Mossy fishing bait for the Black Forest.

The actual mechanics themselves is that you need to start reeling in when a fish bites the line. There is both a visual and audio cue for that happening but both are prone to being easily miss-able due to ambient sound and weather/sea effects. I just try to time my reeling in for when the fish notices the baits and comes in to strike at it.