Yes, you can only dig so deep. Canals are possible, but only if the starting terrain between the two ends stays relatively close to sea level.

There is a maximum of (Iiirc) 8 meters in the up and in the down direction from where the generated terrain starts. Barring tides and waves the water level is at a set depth.

The other issue you will run into is when you are digging terrain while in the water. Once you get down to a level where you are swimming (3 meters) your pick will unequip and you won't be able to re-equip it until you aren't swimming.

In your situation you can do what the actual Vikings did. Build a canal as far as possible and build a dock with a ship at each end and a road connecting the 2 docks with a cart. Its still going to be much faster than sailing all the way around a large island.

If you are material poor when you are doing this you can still make it work with a single ship. When you ground the ship make sure that your inventory has enough open spaces to hold all the resources used to build the boat and that your carrying weight has enough give to carry all the materials (you can do this by fully loading the boat if there are no storage chests nearby). Then stand on the boat, on top of the mast, (make sure that you have autogather turned on <"V" key>) and take your axe to it until it breaks. When you fall into the water the materials used to build the boat will go into your inventory rather than sink and be lost. Every thing that was loaded into the boat's cargo hold will become floating crates (even the stuff that should sink). You can pick up the crates, load them on the cart, go to the other end of your road, rebuild your boat, reload your boat and continue on.