In the swamp they are called sunken crypts, a lot of the spaces are underwater, some make you swim... most make you swim to dead ends but yes you get dungeons under water.

You need a Swamp Key from killing the Elder boss to access it.