Fishing Skill doesn't seem to impact fishing to me. Just completed the set and my skill without the hat is a dizzying... 16

Watching for a bite and hauiling in the line in to make strike is 40% of the battle, the other 60% is stamina management which potions are your friend as it knowing when NOT to pull but drag the begger onto the Land/Boat!