Amongst the million-dollar budgets, high-tech equipment and world-class sailors, one thing is seemingly missing from every campaign in this America’s Cup 36 cycle: a training partner. The training partner has long been the important measuring stick, or control, by which teams evaluate performance and guide their design choices. The Cup is cutthroat, it’s secretive, and teams hardly have any training time in their final boats because they need to squeeze as much design development as possible into their race machines.
But what if a team could create a virtual training partner, in-house? One whose design could be updated in code instead of carbon, and that could learn superhuman boathandling in the cloud instead of on the course? What if they could develop a “digital twin,” by using machine learning (ML), to replicate the elite sailors and the infinite number of decisions they make during a race? In partnership with Emirates Team New Zealand, McKinsey and QuantumBlack, a McKinsey company, built such a digital twin (also known as “the AI agent” or “the bot”) to test hydrofoil designs. They unleashed this bot on ETNZ’s simulator, the same one used by the sailors to train off the water, where it collected and learned from lifetimes’ worth of simulated experience and allowed them to develop and optimize hydrofoil designs nearly 10 times faster. The bot was no match for Burling and crew during initial head-to-head contests, but the powerful AI within eventually bested the America’s Cup defenders.
Over thousands of simulated races in a range of conditions, the bot trains itself to get the best performance out of a given foil design, so then each foil design’s performance can be quantified in terms of synthetic lap time, which includes a range of conditions based on how likely they are to occur. The major benefit of having the digital twin was that the team could test foil iterations “on demand, at speed, and at scale,” says Nicolas Hohn, Chief Data Scientist at QuantumBlack. America’s Cup teams are inherently racing against time, so quick iteration is paramount.
In addition to saving time, using a digital twin results in at least a 95-percent reduction in testing cost per foil design. “The big shift this allows is eliminating the opportunity cost of having the sailors do the testing,” explains Brian Fox, a Senior Partner at McKinsey. While the sailors and design teams are sleeping, meeting, and training, the bot is constantly working: sailing laps in the simulator and trying different parameters.
The 12-month project with McKinsey started with MIke Buckley and Taylor Canfield’s Stars+Stripes campaign about 18 months before the Cup to allow for time to manufacture the team’s final hydrofoils. JB Braun, Director of Design and Engineering for North Sails had an idea to apply expertise in ML and AI to advancing various aspects of the design process using ETNZ’s simulator.
Dan Bernasconi, Head of Design for ETNZ, always has a number of technology companies coming to him with lofty promises. He saw the value immediately but stated that he didn’t think it was feasible. In fact, no one in ETNZ thought it could be achieved, but he gave the McKinsey team access to the simulator with a couple of days of simulator training, and said to come back in four to six weeks with something to show for it.
The bot trains itself by using reinforcement learning, a machine learning technique that’s used for identifying the best sequence of actions in a complex environment. Reinforcement learning mimics how humans learn: by trial and error, but without the forgetting and relearning of the same lessons.
ETNZ’s highly-advanced simulator was crucial to the success of this ML implementation because the simulator acts as the environment and the reinforcement learning algorithm can treat the simulator like a black box, caring only about the inputs and outputs. The bot accumulates experience through its interactions with the simulator, and the resulting rewards (or lack thereof) are used to adjust the weights given to the parameters that govern its behavior.
Understanding the journey of training the bot makes you realize how much the sailors are consciously and subconsciously processing while sailing the AC75s, let alone racing them.
Understanding the journey of training the bot makes you realize how much the sailors are consciously and subconsciously processing while sailing the AC75s, let alone racing them. Hohn illustrates the complexity of sailing and hydrofoiling with the fact that there are four inputs for Formula 1 cars, but the bot has to control 14 different inputs for the AC75. Inputs include the steering, rudder lift, lift on each foil, the mainsail trim, the traveler, and the jib trim, amongst others. Foiling adds that extra dimension and extra degrees of freedom such as ride height and pitch.
This project is one of the most ambitious applications of deep reinforcement learning ever. “What we’ve proved is that it’s possible to take theories out of these controlled environments of videogames and put it into an environment with lots of variables,” Hohn says.
McKinsey established four milestones for the bot: learning to sail upwind, then downwind, then tacks, and finally, jibes. They created a network where there were a thousand bots running in parallel, all learning from each other. “It’s learning at a huge scale and it dramatically reduces the time and cost of the project,” Hohn says. “That’s the power of the cloud.”
There are three main reasons why making the reinforcement learning work for real-world sailing is a massive challenge, however. Let’s start with the incredibly complex dynamics of an AC75. ETNZ’s simulator takes care of figuring out how the boat responds to various inputs (steering, sail trim, foil angle, etc.), but the bot has to learn the fine line between just enough, but not too much. What makes reinforcement learning powerful is that the bot learns to respond with very different “optimal actions” for slight changes in the situation, which ties really well into what makes great sailors. As all competitive racers know, what sets champion sailors apart are lots of tiny changes and knowing how to walk the fine line, rather than a couple of massive differences.
The second main challenge is imperfect information. For example, because the sails are dynamic and change shape with the ever-changing pressure of the wind, it’s hard to know what the optimal input is. Hardly anything in sailing is truly fixed.
The third main challenge is the delayed rewards coupled with loosely defined goals. “There’s no single recipe on how to sail well, so we had to try to understand what we’re optimizing for. That allows us to invent and refine the right reward function that enables the bot to sail as efficiently as possible,” Fox says. In terms of encoding the rewards, they were ultimately solving for VMG. That means that at every time step, the simplistic reward is how far you’ve gone in the direction of the next mark.
“One of the challenges was setting up the reward function because of the tradeoffs between short-term and long-term gains on the course,” Hohn says. For instance, you may need to go into a low mode to get to the next puff, which could be a short-term hit to VMG, but a long-term gain.
With the creation of any simulation in sailing, however there’s the constant question of how well it matches reality. For instance, upon seeing how the bot would perform speed builds after a tack, the team had to determine whether it was an artifact of the simulator, or whether they were onto something. To ensure they didn’t find different optimal foil designs when controlled by the digital twin versus the sailors, the team had to implement real human constraints, such as not moving the traveler, foil flaps, or rudder faster than a human could.
Just as the sailors have their coaches, the machine learning engineers have to “coach” the bot. If you only tell the bot to figure out how to sail a lap in the least amount of time, for example, it’s going to take it a long time to figure that out because it’s like dropping someone who’s never sailed on a boat in the middle of the Pacific and telling them to figure out how to get home. There’s a balance to guiding the bot so it can learn fast and actually converge to a solution, while also not over-coaching it with unfounded assumptions so that the bot can surprise you with unprecedented solutions and new insight.
For example, Hohn explains, early on, the bot figured out they were running the laps all of the same duration, so at the end of the lap it would go head to wind for a few seconds to maximize VMG—like shooting the line at the finish. In order to get accurate outcomes, they had to vary test duration so the bot couldn’t figure it out.
No surprise here: ETNZ’s athletes sail at a very high level. It took a long time for the bot to reach their level. Because the bot sailed the boat differently, the sailors were able to see a new perspective on speeds and maneuverability that’s achievable on the boat, and got inspiration from the bot on new moves, sailing tactics and ways to reduce variability. Reinforcement learning allows for more objective decisions by removing the human bias, such as preconceived notions about how a design change will behave. “Automating part of the design evaluation removes that uncertainty of human inputs,” explains Fox.
Successful development under the accelerated timeline of the America’s Cup is a lot about not getting sidetracked. It’s analogous to guiding the AI bot: you have to give the team, or the bot, enough guidance to stay on track, while simultaneously giving enough leeway to find the gems that lead to an unexpected edge. Any competitive sailor with a creative mind can relate to this internal tug-of-war.
Yet, even with AI, each team must make big-picture decisions about where to focus their precious development efforts. Prioritizing pure straight-line speed, reliability, maneuverability, or anything else, will influence the decision of where to apply AI. It’s then up to the sailors to use the strengths of their boat and style of sailing to their tactical and strategic advantage.
Hohn says what’s most impressive about ETNZ’s team dynamics is that they are “very technical, and have seamless integration between all aspects of the organization. Between the sailors, designers, shore crew, and analysts, there’s excellent information flow and really collaborative development. Helmsman Peter Burling was weighing in on fundamentals and the minutiae of the design.”
The project has been fully handed off to ETNZ’s design team and they can run it off their desktops. So how transferable is this project for future America’s Cups? The answer depends more on whether the same simulation platform can be used than on the boat design. If the boat design was completely changed, they’d have to load a new boat model into the simulator and adapt any key controls that changed, plus adapt some of the rewards and constraints. Training a new bot takes weeks.
What does the future of AI and ML in sailing look like? All sailors can look forward to watching and learning from a bot’s discovery of innovative tactics and techniques. AI and ML for tactics and race strategy will be very important, and ETNZ has already been experimenting with this. We can collect data from many bot vs. bot simulations to put hard numbers behind tactical decisions (even though any one decision has a random component to it because the wind might shift). Imagine saying with certainty the probability of success given your current position. You can expect to see a lot more in this space.
Hohn says he loves using sports as a proving ground for more untraditional technologies. In partnership with Altair, Luna Rossa Prada Pirelli, ETNZ’s Challenger in the America’s Cup, used digital twins to apply analytics and testing during the design process with a focus on the load paths in the hull to refine the internal structure. You can expect all of the teams in the next America’s Cup to apply AI and ML to more aspects of their boat design and race strategy, and push these capabilities in the process.
As for the promise of what this means for industries outside of sailing, reinforcement learning can be applied to any complex process where the control parameters are not easily optimized. Digital twins can be applied to a broad range of industries and with their relevance to product development, manufacturing, supply chains, and asset management, you can expect to see them as one of the top technology trends this year and beyond. For the teams and businesses that start today, well…it’s like winning the start.
Helena Scutt is a 2016 Olympian in the 49erFX Women’s Skiff and has also raced foiling Nacra 17s. She is a mechanical engineer and Moth sailor based in San Francisco.
