Please select your home edition
Edition
Excess Catamarans

Science blog: Machine learning and whale song

by Ann Allen, NOAA Fisheries 3 Nov 2018 02:13 AEDT
A humpback whale surfaces near Maui Island © Hawaiian Islands Humpback Whale National Marine Sanctuary / Jason Moore

What do you do when you have too much data? It's not a question I've ever had to ask myself in my scientific career, and is rarely a problem when working with hard-to-study animals like whales and dolphins.

But this is exactly the question I found myself asking shortly after starting my new job with the Pacific Islands Fisheries Science Center's Cetacean Research Program in Hawaii in 2017. I was describing my job to my father, who is not a scientist, and trying to convey what we want to do.

We conduct research to understand the health and status of whale and dolphin species around U.S. Pacific Islands and across the Pacific Ocean, a VAST area. One tool we use to monitor elusive whales across this large region is a network of seafloor mounted recorders called high-frequency acoustic recording packages (HARPs).

The HARPs have hydrophones (underwater microphones) that can record the calls of whales and dolphins, as well as any environmental or human-made noise. We deploy HARPs for months to years at a time and we have to retrieve them to obtain data.

So far, our network extends to 13 different recordings sites, of which several have been monitored for a decade or more. This means we now have more than 200 TB of data, or almost 170,000 hours of recordings. If you were to sit and listen to all of that audio straight through it would take you more than 19 years. So if we monitor the whales by listening to their calls in the recordings, how in the world do you ever listen to that much data?

After telling my father all of this, he asked me, "Well why don't you just get those Shazam or Google people to help you? They teach computers to recognize human songs, I bet they can teach it whale song."

I paused. "Well, because it's just not that simple..."

"That's just not how you..."

"Because people have been..."

"Um well y'know, I don't know."

Thus challenged by my dad, I figured I'd find out if those Google people would be interested in teaching their computers to speak whale.

A friend who works for Google put me in contact with their Artificial Intelligence group. They responded almost immediately and enthusiastically; an ocean's worth of data sounded like it would fit in well with their current projects. They already have a lot of machine-learning techniques they've developed for other purposes that they would be excited to put to use for conservation. Machine learning attempts to mimic the learning process of the human brain by teaching a computer to carry out a task, rather than programming it to do the task step by step.

Automatic detection of whale calls isn't a new approach, but designing a good detector can be quite difficult. The ocean is a noisy place, with lots of sounds from wind, waves, fish, and even the recording equipment itself. And humans contribute an ever increasing amount of noise to the ocean, from things like container ships, oil and gas prospecting, military activities, and construction. The whale vocalizations themselves can also sound very different depending on how far away the whale is from the recorder, how it echoes off of the ocean floor and surface, and whether it has a local variation, or dialect, so to speak.

While some whales, such as blue whales and fin whales, make consistent, stereotyped calls that are relatively straightforward to recognize once you've seen some examples, humpback whales have vocalizations that are extremely variable. These calls make up phrases and themes that form songs. The songs change over time, as the whales incorporate new notes or swap the order of phrases, and many populations of humpbacks sing different songs. So how do you teach a computer to recognize whale song when the song keeps changing?

Researchers have been trying to solve this problem for quite a while. So far all the methods developed come with caveats: poor detection rates, a lot of user input to fine tune the detector for each call type and situation, or a high rate of false detections that require someone to go through by hand and pick out the correct detections from the bad ones.

However, there have been a lot of advancements in the last few years in artificial intelligence, and deep-machine learning. You may have noticed that your phone is much better about not making tie bows, I mean typos, when you use voice recognition these days. This is due to advances in machine learning, some made at Google. So with this in mind I figured I would throw Google in the deep end and see if they could come up with a solution to the complex humpback song problem. I gave them a subset of our audio data, with a collection of 'training labels', or humpback whale calls that are marked by a human (me). This training data is how the computer learns what humpback calls look like.

And it turns out that Google's computers are pretty good at recognizing humpback song! In less time than I thought possible (practically gale force speed by academic standards), they sent me timestamps of humpback whale song in our data. The computer isn't perfect yet, we still need to teach it to better recognize different kinds of noise from sites that didn't have as much training data, but it is already doing far better than expected.

There is so much new, important information we can learn from just knowing when there is humpback whale song throughout our dataset. In the Pacific Islands region, very little is known about humpback whale presence, seasonality, daily singing behavior, or population structure outside of the main Hawaiian Islands. We do not even know if humpback whales ever journey to some of the smaller and more remote uninhabited islands. With this data we can start to address these questions. Additionally, because our dataset spans more than a decade, knowing when and where humpback whales are singing will give us information on whether or not the animals have changed their distribution over the years, especially in relation to increasing human activity in some areas.

The possibilities don't stop there. We're now interested in teaching the computer to find other species of whales and dolphins in our recordings. The hope is that if we give the computer enough examples of different whale calls it will learn to generally recognize 'whale' sounds and get better at distinguishing between whale species and also between 'whale' and 'ship' or other noise. My hope is that in the future Google will provide us with a tool that scientists can directly use. A tool that we can provide new data, from a new location, and training labels for a new type of whale call, and it can then tell us where those calls are in the rest of the data.

So, if you're a scientist with acoustic data and labelled whale calls and want to help teach Google how to speak whale, whale then give us a call!

Check out more details of the story from the Google side:

Related Articles

Biotherm wins the Ocean Race Europe
For Paul, this was most satisfying of wins in a race that he planned for and targeted for many years Paul Meilhat and his crew on Biotherm are the runaway winners of the second edition of The Ocean Race Europe after adding their fourth leg win in five starts early this morning. Posted today at 12:36 pm
Holcim-PRB claims 2nd place in Boka Bay
After memorable comeback in The Ocean Race Europe Leg 5 The Holcim-PRB crew claimed an outstanding second place on the fifth and final leg of The Ocean Race Europe, navigating a course full of twists and turns. Posted today at 8:59 am
Biotherm win final leg into Montenegro
To seal dominant overall victory in The Ocean Race Europe 2025 Paul Meilhat's French-flagged IMOCA Biotherm has won the fifth and final stage of The Ocean Race Europe - the 1,600-nautical mile leg from Genova, Italy to Boka Bay in Montenegro - and in doing so has confirmed a spectacular overall win. Posted today at 5:43 am
Team Malizia Third in Boka Bay
Clinching Podium Finish in Final Leg of The Ocean Race Europe 2025 After a tight battle for the podium on the fifth and final stage of The Ocean Race Europe 2025, Team Malizia crossed the finish line in Boka Bay, Montenegro, early this morning to claim third place and five points. Posted today at 5:33 am
Sydney International On-Water Boat Show Preview
Set to deliver a premium festival experience this November With just eight weeks to go, the Sydney International On-Water Boat Show is shaping up as a must-attend celebration of Australia's marine lifestyle. Posted today at 5:20 am
Olympian Wins Match Race Nationals
Nia Jerwood victorious in the Australian Women's National Match Racing Championship Two-time Olympian Nia Jerwood entered the event with little match racing experience. She proved that she has three valuable assets however: She is fast, has extraordinary powers of determination and concentration and is a very quick learner. Posted today at 5:08 am
Biggest ever OK Dinghy Worlds Opens in Italy
More than 200 OK Dinghy sailors took to the water for the practice race at Lake Garda Never, in all of history, have so many OK Dinghies gathered in one place at the same time and gone sailing. On Sunday, at Lake Garda, for the first time ever, more than 200 OK Dinghy sailors took to the water for the practice race. Posted today at 4:46 am
Recycled. Reborn. Ready for Adventure.
Henri-Lloyd transforming marine waste into purpose-built performance Crafted from NetPlus ripstop with midweight Repreve insulation that delivers incredible warm. Built for coastal weather, shifting winds, and the rhythm of the sea. Transforming marine waste into purpose-built performance. Posted on 14 Sep
La Solitaire du Figaro Paprec Leg 2 start tomorrow
After 486 miles of a modified course, the Spanish coast will come into sight On Monday, September 15 at 2.45pm, the 34 solo skippers will set course southwards towards the port of Vigo, Spain. A leg already shaping up to be demanding, with a depression forcing the race committee to postpone the start by 24 hours. Posted on 14 Sep
XR 41 Documentary: Chasing Gold at the ORC Worlds
Watch this great video which goes inside the X-Yachts Racing Team Episode 3 of the XR 41 documentary series is now online. Titled "Chasing Gold at the ORC Worlds - Inside the X-Yachts Racing Team", this episode follows our racing team through an intense campaign at the 2025 ORC World Championship in Tallinn. Posted on 14 Sep
Hyde Sails 2024 - One DesignAllen Dynamic 40 FooterNorth Sails Loft 57 Podcast