Long before this publication launched to chronicle the development of self-driving technology and the industry from our vantage point at Argo AI, the term “ground truth” was well-established in computer science, artificial intelligence, and, for centuries before those fields existed, cartography.
As the Oxford Reference defines it, “ground truth” refers to “the facts that are found when a location shown on a map, air photograph, or satellite image is checked on the ground, as validation.”
From these descriptions, establishing ground truth sounds deceptively simple at first: facts that can be confirmed from observations on the ground or out in the physical world.
The loftiest definition of ground truth is that it is the “actual answer.” Other affectionate terms might be “the truth,” or “absolute truth,” “correct answer,” “what is,” or any other word or phrase that describes “the answer.” Ground truth is the barometer by which you measure other data, the gold standard, the lines on your grid, or perhaps, if you’re feeling poetic, the permanent, immobile rocks against which the wind blows.
But thinking through these definitions and descriptions, many questions arise.
Just who is responsible for observing the facts on the ground: a single individual, a team, a machine? How can we ensure they will be accurate?
What instruments and equipment should we use to measure and record the truth? If we use a ruler to measure the length of something, how can we be certain the ruler is accurate? What if the ruler is mislabeled or wrong? What if our hands are shaky while we measure, or we misplace the ruler? When are we making our observations? What time of day, and how often, and does this change any of our results? How finely must we measure: down to the inch, centimeter, nanometer, or subatomic level? Are approximations good enough to establish ground truth?
If we were back in Ancient Greece, we could simply ask an oracle, that is, the one who knows the truth. But we are not in Ancient Greece, and even then, how would the oracle determine what’s true?
This series of questions sounds almost philosophical or existential, but is actually of huge practical import when designing autonomous technology you can trust — in our case, the Argo Self-Driving System (SDS), the software and hardware system behind our growing global fleet of autonomous vehicles. The Argo SDS is designed to safely drive from one location to another in our operational areas, and to interact safely with anyone and any traffic situation it may encounter along the way.
In designing, building, testing, and deploying our SDS on public roads, we need to establish many different types of ground truth, and all of them must be accurate enough to enable our self-driving vehicles to move safely. We need to ensure that all the data our cars are referencing, gathering, and reporting to the SDS is correct, up-to-date, comprehensive, and precise enough to accomplish the task of safe self-driving.
So, how does ground truth enable us to do this? First and foremost, our SDS uses a collection of sensors mounted outside of each one of our self-driving test vehicles to detect what’s happening 360 degrees around it. These sensors are calibrated and checked continuously to ensure they are measuring and detecting the world properly. This sensor calibration is done by human engineers using physical targets, test benches, and other equipment, and it occurs regularly in our vehicle depots in eight cities and counting, and on our closed courses in Western Pennsylvania and Munich, Germany.
Not everything our SDS sensors detect is relevant or helpful for the task of safe self-driving. The shape of the clouds in the sky is not especially relevant to how we drive autonomously on the ground, for example. Ground truth helps us remove this kind of data, which we call “noise.” Establishing what data our SDS should not use as a ground truth is as important as establishing what it should use.
We’ve seen this very important point demonstrated in the real world just last year, when a human driver reported that the advanced driver assistance software in his car confused the moon for a traffic light and slowed down accordingly.
To avoid this specific type of issue, the Argo SDS uses lidar, a type of sensor that measures the time it takes for lasers to bounce off surrounding objects to estimate their distance, motion, and speed. Lidar helps our SDS establish an accurate representation of the physical objects around it, and recognize that a cloud is not an object it needs to avoid, or that the moon is not a traffic light it needs to obey.
Noise occurs in other areas of self-driving detection, data gathering, and analysis, and we use similar processes to remove it — writing software to identify important data and to discard the unimportant data, which goes through various levels of human reviews to ensure it is working properly.
Another way in which the Argo SDS relies upon ground truth is in localization. That is: how do our self-driving cars know where they are in the world?
Our SDS localization system incorporates information from a self-driving car’s external cameras and lidar sensors, its tires, its engine revolutions per minute (RPM), inertial measurement unit sensors, position as recorded by global positioning satellites, and more. While driving, the SDS compares lidar imagery of the world around it to previously recorded lidar data stored in our high-resolution local maps, and uses this, plus inertial measurement sensors and RPM to determine the car’s precise location on the map.
Our localization system might say that for a given Argo self-driving car, we are at position XYZ in latitude and longitude, at orientation ABC, moving at speed S, rotating at rotational speed R around axis XYZ.
But those are all estimates: the truth is that we might be off a little in all of those values. How would we know?
Part of our knowledge comes from our hardware engineers and their continuous testing and evaluation of the vehicles. Another big part of it comes through comparison. On most weekdays, teams of highly-trained, certified, and independently audited, full-time Argo employees called Test Specialists ride in the front of our autonomous test vehicles throughout eight cities in the U.S. and Germany.
These rides occur multiple times per day for hours at a time using several different types of test vehicles. During their test drives, the Test Specialists monitor the safety and performance of our SDS as it operates autonomously, but also gather data in the form of camera, radar, and lidar imaging, and using their own eyes and ears. Put another way: our Test Specialists measure the “ground truth” of whether or not our AVs are staying within their lanes, stopping at the stop lines often painted on the roadways, stopping outside of the crosswalks to allow pedestrians to go safely across the street. The Test Specialists measure this ground truth to the accuracy of being consistent with the law and with social norms in the city in which they drive.
Argo’s Test Specialists also help perform mapping missions in each of our test cities. During these missions, the vehicle sensors gather data on the roadway geometry, road signs, traffic signals, road markings and structures around the roadway. This data is used to create richly detailed, 3D, high-resolution digital maps of all of our driving areas. These maps are the “ground truth” of what is really out there…or at least was out there when we mapped the roadway. We double- and triple-check the data to get this ground truth.
These maps include far more information than a typical consumer mapping app on your smartphone, such as the specific roadway markings that delineate left-turn and right-turn only lanes, and lane lines themselves. Argo’s maps are updated multiple times each day and can account for sudden and temporary changes to the roads in each city, such as areas blocked off due to construction. Updated maps are pushed to our test vehicles every day before they begin their test driving shifts.
Test Specialists who sit in the passenger seat of our vehicles help to annotate the maps by observing changes to the roadway environment during their test drives and recording them in note-taking software on a laptop. Another step in acquiring, maintaining, and updating ground truth.
Each time an Argo car goes out and gathers data about its route, we compare it to a previously established series of “ground truth” parameters gathered on prior test rides. We run test drives at different times during the day and night so that our system can account for variances in the roadway conditions and traffic that take place throughout each day.
Another form of ground truth trains our SDS’s perception system to recognize other road users and objects on the road. Argo’s machine learning software algorithms analyze human-labeled video footage and use it to learn the physical qualities of different categories of objects — pedestrian, bicyclist, other vehicle, for example — and how to distinguish said categories apart from one another. Importantly, Argo does not apply any facial recognition or other personally identifying technology to the images collected. The “ground truth” needed here is only the object type, size, and speed.
Therefore, our perception system relies upon a set of continuously updated ground truths that are established from real-world test drives and the intelligence of human employees and analysts.
Outside of our perception system, we use different ground truth values to evaluate the performance of all the other subsystems that make up our SDS. Among them are motion forecasting, motion planning, and controls.
These programs all rely on access to accurate data about the speed, number and types of objects around our cars, and the probability of how those objects will move. That “accurate data” is yet another instance of ground truth: accurately annotating real data with the ground-truth interpretation, accurate enough to detect driving errors.
Ultimately, establishing ground truth for self-driving vehicles is a continuous, never-ending process: when it comes to the driving environment, some aspects like signage and environmental features are typically fixed for long periods of time, but others, such as the specific actors on the road, change often. Whatever the rate at which it changes, verifying correct operation of self-driving vehicles requires vast amounts of ground truth data.
That’s why regularly mapping and having people check and recheck all the data and sensors our SDS relies on is so important.
Far from being immutable, ground truth for self-driving vehicles is a dynamic, ever-varying set of values. Even the supposedly permanent rocks change and move over time, as any geologist will tell you.
What is the truth if it’s always changing? While philosophers debate that, our practical answer is actually quite simple. : we do so frequently and repeatedly, using a combination of human intelligence, artificial intelligence, structured testing procedures and data, and by comparing the data we’ve gathered across time and place.
