Technically speaking, you can't simply pass by them usually since they ONLY appear first to then come up and do their roar, but sometimes if you are going fast enough, you actually can go right past them before they even roar and likely not even hear it xD.
Unless they spawn right in front of the boat, which has happened to me twice.