How Argo AI Engineers ‘Follow the Sun’ With a Global Approach to Autonomous Vehicle Testing
In the ultra-competitive world of autonomous vehicle technology development, every hour counts.
With the launch of a new test track and the deployment of a test engineering team in Munich, Germany, Argo AI is developing a “follow the sun” strategy that virtually doubles the amount of productive hours every day. The whole process will start with the rising sun in the Bavarian capital and continue on the other side of the Atlantic—in Pittsburgh, to be precise—as the workday begins stateside. When the American team’s day is done, that data will then be sent back across the ocean to the Munich team, which will wake up to find raw data ready to crunch, test, and load onto vehicles in preparation for the next day of track testing.
This setup allows for a significant advantage over Argo’s competitors: Namely, a near-24/7 development cycle that helps speed the development of software and may ultimately help bring products to market faster. Stepping down from the cosmic perspective, how can it work in practice?
“At midnight eastern time every night, there’s a cut of software called a ‘release candidate’ that’s automatically generated,” explains Argo Staff Test Engineer Michael Pacilio. “There could be up to 10 million simulation scenarios that run against that build of software.” All of those scenarios land in the Munich team’s queue at 6 am German time—“a few hours later, the team shows up and can start digesting that virtual test,” says Pacilio.
That includes looking for “build issues,” anomalies in the software that are revealed when the release candidate has been run on a test bench—a hardware setup that acts like a car. Then the data gets loaded on a test vehicle, and the testing starts on Argo’s new, 9-hectare test track at the Munich airport. When the team’s U.S. counterparts arrive at work at 6am eastern time, they can pick up working on any test results as they stream in from Munich.
“By the time the U.S. team is ready to start on the track, the software will have already gone through 4-6 hours of testing a potential repair,” Pacilio said. “That saves the whole process an inordinate amount of time in getting the software built, tested, and ready for the road, which, ultimately, is the whole point.”
The two teams work in close coordination, with the same management structure, facilitating nearly 18 hours of testing every day, which will increase as Argo expands its operations to test at night. “It’s not only ‘follow the sun’, it’s a sequential and parallel workforce across the ocean,” explains Thomas Bock, who runs the company’s fleet operations for Europe. “So far, we are in startup mode, so we are establishing those data passes, and we are making progress every day as we learn this global scaling.”
With the Munich test track up and running, which allows the engineers to tailor testing to unique European road scenarios, the team is constantly collecting data each day, and doing training there as well. “We’re leveraging the full track and extending the capabilities every day, with more lanes, traffic lights, containers to stand in for buildings, and so on, which help prepare our vehicles for the complexity of European roads on which we’ll be testing,” Bock says.
For example, bicycle lanes and pedestrian crossings are different in Europe than in the United States, as is pedestrian and bicyclist behavior. Traffic lights differ in style between the regions and are located in a different position relative to the intersection, with the light moving outside of a human driver’s field of vision earlier when crossing a European intersection. Traffic circles and bus lane configurations are also distinguishable, as are how cars are parked, relative to the traffic lanes. All of these variances can be simulated for the vehicles on the Munich test track, and analyzed using the streamlined processing power of the teams on both sides of the Atlantic.
Bock points out that beyond the advantage of faster speed of development, such a strategy results in additional capacities. “We can extend our data processing capabilities by leveraging the U.S. teams, which means we have roughly 150 percent capacity, just by using our internal resources.”
This setup allows the entire Argo test engineering team to adapt more quickly and troubleshoot problems, and be much more responsive when it comes to requests. “We have to support multiple platforms and two major partners”—Argo’s automaker investors Ford and Volkswagen—“with markets in Germany and the U.S., with different traffic conditions, right of way rules and so on,” Bock says. “We have to be able to test those critical points nearly 24/7, and the faster we can do that, the faster we can get the vehicles safely on public roads, and the faster we get to a commercial release.”
Pacilio points out that with the winter fast approaching, having a tight coupling with the Munich team means the number of daylight hours can be essentially doubled. “That alone is a huge benefit to the entire testing operation, since we design test operations specific to day and for night,” he says.
In addition, having team management centered around solving the same challenges is crucial to the strategy’s success, says Pacilio. “I can send instructions to Munich to set up their traffic light configuration the same way they did in the US, which means we can use the same test documentation. That means there’s continuation and reduces confusion when you’re transferring that data.”
From Pacillo’s perspective, being able to extend the testing day is a no-brainer, and is on its way to exceeding expectations. “I am a tiny bit gleeful at how well-integrated the team actually is, and how well the engineers in Munich have picked up the mantle of doing this initial bench level testing,” he says. “That’s grinding through code and mountains of test results every day. I have to give the team over there a lot of credit.”
Pretty soon Argo employees will be joining a physical implementation of the “follow the sun” strategy, thanks to cross-training programs whereby members of the Munich team travel to the U.S., and vice versa. “It’s all about generating that type of enthusiasm, around a singular objective,” Pacilio said. “We’re going to put an autonomous vehicle on the road safely, and make it profitable. Sharing that objective makes it easy for us to work together.”