October 9, 2019
How Salesforce Einstein is Helping Track (and Protect) Great White Sharks in the Wild
By Nick Johnson
Marine biologists and AI researchers have teamed up to develop new shark surveillance tools.
Over the past four years, scientists have noted an increase in great white sightings off the California coast — particularly 5- to 10-foot-long juveniles. There are many theories as to why this is: climate change; the effects of 25 years of protected species status in California; banning gill nets in nearshore waters that indiscriminately kill marine life, including sharks; an increase in the population of seals and sea lions, which are a key food source for adult sharks, to name a few potential factors.
But recent sightings near popular surfing areas in Santa Barbara were enough to raise the interest and concerns in the local community. McCauley teamed up with Salesforce Director of Research Michael “MJ” Jones, Salesforce Chief Scientist Richard Socher and San Diego State Associate Professor of Computer Science Xiaobai Liu to devise a new way to identify the presence of sharks in coastal waters using artificial intelligence.
The Salesforce Research team theorized they could train the Einstein Vision algorithm — ordinarily used for visual search, brand detection, and product identification — to process drone video and recognize the unique attributes of sharks, thereby identifying them without the need of physically tagging them. Working with McCauley’s team, they devised an innovative workflow for video capture and processing, all without ever touching the water. They dubbed it “Project SharkEye.”
Images of white sharks spotted off the coast were fed into an AI model to train it to identify them.
The first step was collecting huge quantities of video data off the coast through drone technology. That video would be the basis for building the AI models used to identify a great white shark. McCauley and his team worked with UCSB students, who piloted drone flights off the beach, filming a 10-mile stretch of the coastline.
“The benefit of drones is that you’ve got an eye in the sky to see more of the ocean,” explains McCauley. “It’s super cheap to fly up and down the beach. That’s the easy part. But what comes after is where AI becomes really valuable. Every hour of video footage requires an hour of a video analyst’s time. If you’re trying to gain insights in real time, that’s not efficient. That’s where computer vision really comes into play.”
Pilots from UC Santa Barbara fly drones regularly on local beaches in search of great white sharks.
With sufficient video data collected, the Salesforce Einstein team worked with the marine biologist teams at UCSB to train the algorithm on a number of factors. It had to learn, for instance, to differentiate between shark species, such as a leopard shark (which is harmless to humans) and a great white. It had to rule out non-shark objects, such as a large floating piece of kelp with a shark-like shape, and determine whether a sighting was a new shark or a second sighting of a shark already counted. With each sighting, the algorithm got smarter.
“The way we explain it is that it’s almost exactly like how kids learn,” says Einstein Project Manager Zineb Laraki, one of the project’s leads. “You tell them, ‘That’s a cat.’ Eventually, kids can generalize and start to understand the concept of a cat — what it looks like, how it moves, predicting what it will do based on watching it. It’s the same thing with the algorithm. You show it examples. You let it make a determination. Right or wrong, you give it feedback.”
Once trained, the AI can differentiate white sharks from other objects and harmless sharks.
Beyond the technology, the Salesforce and BOI teams wanted to get buy-in from a community that bases its livelihood on the coast, such as surf shops and schools, local businesses, and public safety officials. That’s why the extended plan is to offer an adapted version of the Salesforce Field Service Lightning mobile app — typically used for onsite job management, providing real-time collaboration and access to information such as scheduling and knowledge bases — to local stakeholders like lifeguards, surf school teachers, and marine biologists. Those “field workers” will be able to access the collected information, such as how many sharks were spotted at any given time and where. The rationale is that by providing such tools that involve the community in the process, they’ll trust the AI technology as a solution.
“This application is unique because we’ve had the privilege of building it hand in hand with our community partners,” explains McCauley. “For us, it was a fantastic opportunity to apply AI to not only capture data of value to shark researchers, but to also be able to try to be helpful to our community by sharing that data.”
It’s also about understanding a traditionally misunderstood species — where they’re swimming and why, their behavior and feeding patterns. If we’re able to give these creatures their space and predict when they might come into contact with people, McCauley says, we’ll be better prepared to keep beaches safe, too. Not to mention the sharks, who are still a “vulnerable” species — one step up from “endangered” status. The end use case of such technology isn’t just a practical use of AI — although it is that, and an undeniably cool one. It’s also a way to bring technology to a place it hasn’t traditionally been: the beach.
“Lifeguards do a superb job of protecting all of us when we head to the beach,” says McCauley. “But they are constantly on the lookout for next-gen tools and technologies to help them better assess the situation at their beaches. AI could very well be one of those new tools.”
Project SharkEye promotes safe sharing of the oceans between people and wildlife.
Perhaps in the near future, McCauley suggests, lifeguards could be aided by AI-powered autonomous drones installed at every tower, and those drones could make regular flights and deliver data back to the tower. That opens up a whole universe of knowledge — for instance, what is the status report is on the day for shark sightings? Are the waters pollution free? Are there any dangerous riptides forming?
“AI is doing amazing, audacious things and inspiring customers to take that leap,” says Jones. “But there’s lots of reluctance to actually dream when it comes to AI. Real-world applications like this one can help drive actual impact in the world.”
To find out more about Salesforce’s AI research, check out https://einstein.ai/