The Secret to Scaling Self-Driving Technology: Testing in Multiple Cities
Toggle Filter Camera
At Argo AI, we want self-driving technology to safely serve millions of people around the world. That’s why we’re taking a unique approach to testing, one that will give us an ability to scale our technology in a safer and more rapid way so we can expand to cities across the globe. The multi-city approach to developing our self-driving system today, combined with our automaker partners’ expertise in manufacturing and operations, can best deliver on the potential that we all see for this technology in the future.
In order to build a capable and trustworthy self-driving system and expand Argo self-driving technology to many cities, we are testing on public roads in multiple challenging cities now. This exposes our system to a diverse set of complex scenarios, behaviors, and imagery that forms the basis from which we train the self-driving system. That diversity comes from the uniqueness of how humans and their movements “appear” to the vehicle’s sensors in terms of reflectivity and geometry — their differences in shape, physical characteristics, and behavior, including the way they drive, walk, ride, and make decisions on when to yield. Furthermore, the variety of urban infrastructure that surrounds our vehicles out on the road produces a significant number of complex situations to which our self-driving technology must respond.
We currently have test fleets operating daily in Pittsburgh, Detroit, Palo Alto, Washington D.C., Miami, and Austin — plus Munich starting in 2021 — constituting what may be the largest, most diverse active urban-testing footprint of any self-driving vehicle developer.
If the initial selection of cities is done well, then each additional city will be similar to where our cars have already operated. Interactions with vehicles, pedestrians, and bicyclists will look similar to scenarios we’ve already encountered. The commitment to testing in multiple cities now enables faster deployment later, because it provides a broad foundation and allows for additional cities to be selected and tested to quickly “fill in the gaps” as needed. For example, when Argo AI began testing operations in our sixth location, Austin, late last year, we were up and running in autonomous mode within a few weeks of having test vehicles on the ground. In spite of this sped up deployment, we experienced comparable performance to what we were seeing in other cities.
That said, just as mastering simple environments does not necessarily prepare a self-driving system for dense, difficult urban environments, focusing on a single complex urban environment does not fully prepare the system to enter other complex urban environments very quickly. Every city is different in unique and challenging ways.
What counts when training a self-driving system is not just the number of interactions but also the richness and diversity of interactions it experiences with vehicles, pedestrians, bicycles, scooters, and more. A self-driving vehicle that operates safely for many miles on the same roads but never encounters a cyclist weaving through traffic, or a pedestrian walking outside of a crosswalk — or multiples of each at the same time — may make poor decisions when confronted with these difficult interactions. By testing in six unique cities, we ensure that the system encounters diverse scenarios with a wide variety of road actors at high frequency.
Plus, each new city has its own unique culture, topography, climate, traffic patterns, and driving behaviors. In order to operate, self-driving vehicles need to understand what road actors are doing, and what they could potentially do in any given location. A more diverse set of experiences better prepares the system for these possibilities, because it has encountered and learned from past circumstances. Simply put, a diversity of complexity makes the system smarter and safer.
What counts when training a self-driving system is not just the number of interactions but also the richness and diversity of interactions.
Of course, it’s natural to think that we could achieve scale another way: By collecting data from sensors loaded onto vehicles that are equipped with driver-assist technologies like camera-based lane-keeping systems — many of which are on the road today. However, the truth is that the sensors used for driver-assist features are of limited value for developing an SAE Level 4-capable self-driving system. The resulting data contains fewer pixels, and the cameras are not in the same position, making it impractical to train and validate a Level 4 self-driving vehicle, which has a far richer sensing ability.
Additionally, while diversity of data is critical, too much data can be a hindrance because the key to success is the ability to use the data for repetitive testing and evaluation in simulation. While data captured from a driver-assist system can serve as inspiration for simulated test scenarios, it lacks the telemetry information required to develop predictions of behavior and test what the self-driving system would have done. Such information is critical to understanding and refining algorithm performance.
Our process is focused on balancing diversity of location with rigorous testing to deeply understand the details in each. Our intent is to dig deep into the expectations for road behavior in each city we select. Hence, our initial challenge in identifying the set of cities that would expose our vehicles to the widest possible variety of interactions and behaviors, all while keeping safety top of mind.
We also test in multiple cities in order to avoid “overfitting,” a common Machine Learning problem when what works well in one environment does not translate to another. (This is a bit like a retailer using a one-size-fits-all strategy for selling clothing, and we all have experienced how that turns out.) Instead, we’re taking a more tailored approach in development of our self-driving system
We don’t want to create a self-driving vehicle that operates well in Miami traffic, but risks causing issues in other locations. For instance, in Miami, due to both state regulations and local behavior, drivers often only yield to pedestrians in crosswalks (without signage mandating otherwise) when the pedestrian is within the vehicle’s lane of travel. If a driver yields to pedestrians on the other side of the road, other drivers may get upset, creating an aggressive response or other unsafe situation. But this custom is not the same in Washington D.C. or California, where drivers yield to pedestrians anywhere in the crosswalk.
Stay in touch.
The system also needs to understand that traffic infrastructure can vary widely. We want our self-driving system to be able to identify and handle the vertical traffic lights in one city as well as the horizontal ones in another.
To avoid overfitting, we have carefully selected our test cities to provide our system with a broad array of challenges that are representative of what might be encountered in cities around the world:
- Pittsburgh has its share of idiosyncrasies: hills, narrow streets, numerous bridges, and seemingly more five-way or otherwise quirky intersections than most other cities.
- Detroit features wide lanes and boulevards, shared center-turn lanes, as well as four-season weather that can cause steam and vapor from exhaust systems and street vents.
- Palo Alto sees a wide variety of walkers, runners, and cyclists, including those traveling in groups, and unique road infrastructure such as Botts’ Dots.
- Miami has modern infrastructure and is relatively flat, but it is jammed with the full spectrum of actors: pedestrians, bikes, mopeds, scooters, rollerbladers, hoverboards, cars, buses, and trucks, all of which is operated by a multitude of international drivers.
- Austin’s streets include lots and lots of scooters, which are used by riders of widely varying skill levels.
- Washington, D.C has heavy traffic and some of the most complex traffic-control measures found anywhere — including complex roundabouts, in addition to the occasional bald eagle flying down the street (really!).
Understanding all of the variables in different driving environments is difficult. But testing in multiple cities now will ultimately enable us to expand into others faster, reducing the time it will take for our partners to deploy services across the globe. While the initial commercialization of self-driving technology may be on a street-by-street, block-by-block basis, the rubber will really meet the road with our ability to expand city-to-city.
A multi-city development approach delivers the diversity of complexity required to prepare our self-driving system for the real world — because that’s the world we live in.
And, our pursuit of scalability is the only way we know how to deliver on massive promise behind self-driving technology. That pursuit, in combination with our automaker partners’ track records for scalable manufacturing, fleet management, and customer service operations, is bringing our vision into clearer focus every day.