Hit enter to search or ESC to close

What Do Self-Driving Car Testers Share With Rally Car Racers? The Art of ‘Commentary Driving’

Toggle Filter Camera

Whenever I’m driving with my mom in the passenger seat, there is a constant monologue of instruction coming out of her: Watch that car ahead of you. Do you see that pedestrian? Go slow, the speed limit just dropped to 35!

Yes, mother, I see it all. I’m just not using my outside voice.

But maybe I should be.

The process of verbalizing observations and upcoming obstacles while behind the wheel is called “commentary driving.” As you likely remember from teenagehood, this technique is often used by driver’s education instructors for helping new drivers to be mindful of their surroundings. At a more extreme level, commentary driving is used by rally car racers to help them navigate the dangerous terrains of off-road races such as the Dakar Rally and Baja 1000.

In the world of autonomous vehicles, however, commentary driving plays a different  role, and its effects are helping shape the very future of the technology. 

How it works

Self-driving company Argo AI utilizes commentary driving to aid in the safety of testing autonomous vehicles on public roads. Currently, the development phase of self-driving technology involves humans in the front seats. Pairs of Test Specialists, or TSs, accompany Argo self-driving test vehicles, ready to take over if necessary. Argo Training Manager Natalie Ferrer helps teach commentary driving, in which Test Specialists commenting from the left seat, or driver’s position, calmly call out everything from changing traffic conditions to potential obstacles and hazards, as well as passing pedestrians, trees that could obstruct visibility, a pothole, a construction zone, or a stalled car. 

(Listen to this audio clip of Argo TSs Tre Davis and Jackworth Smith, on a recent drive in Pittsburgh’s Strip District neighborhood, to hear commentary driving in action.)

Meanwhile a TS in the right (passenger) seat monitors the self-driving system on a laptop, comparing human commentary to the on-screen models. That redundancy helps ensure TSs are aware of everything the vehicle will need to detect, and that they’re prepared to take manual control if needed.

Director of Special Operations and Baja 1000 veteran Alex Roy likens the left seat, right seat relationship to the “symbiotic collaboration” of a driver and navigator on a rally team. When they’re hurtling through an off-road desert course at high speeds, the two have to be perfectly synced. “A race cannot be won in a rally car unless both people understand each other in executing their tasks,” Roy says.

But unlike a rally team, the Test Specialists in a self-driving development vehicle don’t have a marked playbook. “While they know the terrain and geography, there is an enormous amount of observation required to monitor the road ahead.”

Retraining the brain

To trainers like Ferrer, commentary driving isn’t just how partners stay connected — it’s a way of improving situational awareness by reteaching their brains how to drive. It also adds another layer of safety by keeping the TSs alert. The technique, which is taught to all Test Specialists over a four-week training period, “makes you slow your brain and the processes down, forcing you to be more vigilant, more safe, more in the present moment,” she says. Take, for example, calling out a stop sign—stating the presence of a stop sign up ahead prompts the driver to evaluate the next steps and brake early enough for a smooth stop at the proper location.

Argo AI trainers use the “SEE” method, a process borrowed from motorcycle-safety courses, which stands for Search, Evaluate and Execute. During the Searching phase, Test Specialists use another acronym, this one invented by Argo, so they know what they’re calling out: CLEAR, or Crosswalks, Lights and Lane Laws, Environments, Actors, and Routes. Busy city streets often have dozens of people and situations that might fit into these categories. So which of these takes precedence? 

In essence, a good TS is vigilant, prioritizes quickly, trusts the vehicle when appropriate and takes over when they deem appropriate for the situation. That starts in the classroom with candidates watching situational videos and commenting to instructors, then hearing how the instructor might have handled the same scene. If there are differences, the trainee and instructor discuss why. For example, a trainee might call out the existence of a parking lot but miss a traffic-light change from green to yellow. In that instance, Ferrer might say: “That light just changed color, and that takes priority over saying that the parking lot to your right is empty.”

High standards

But why does all this matter? Because accurate commentary helps ensure the self-driving system behaves appropriately. Every manual takeover, or disengagement, is logged into the vehicle’s trip data, which is then reviewed by an Argo data analysis team. This step triggers a rigorous software-updating process that, broadly speaking, involves virtual testing, reprogramming the system, and plugging revised code back into the testing cycle. This is what Argo AI calls a “continuous cycle of testing,” as every trip’s log provides an opportunity to improve the software. 

So, while one purpose of commentary is to maintain TS mindfulness, another, equally important purpose is to verify, on a second-to-second basis, that the self-driving system is constantly improving. 

In this way, commentary driving is so much more than just acknowledging every pedestrian or red light that a TS (or my mother) sees. It’s about seeing the world in all its unpredictability and randomness, and responding with confidence.

To learn more about how Argo tests self-driving technology safely on public roads, as well as everything else related to the company’s safety protocols, check out Argo’s safety report.

Choose your lane

How Autonomous Vehicles Distinguish Between Bicycles and People Who Ride Them

How Autonomous Vehicles Distinguish Between Bikes and People

When it comes to how autonomous vehicles see the world, humans come first, literally. Autonomous vehicles (AVs), like the kind operated by Pittsburgh-based Argo AI, use Machine Learning to detect and classify the objects in their surroundings, identifying people...
Why The League of American Bicyclists is optimistic about autonomous vehicles

Why a Leading Cycling Advocacy Group Is Optimistic About Autonomous Vehicles

As autonomous vehicle use grows, AV companies and the League of American Bicyclists are collaborating on how to ensure cyclists and motorists can share the roads safely, even if the “motorist” is artificial intelligence software. As part of the...

Self-Driving Is Arriving Right On Time. Just Like Ice Cream Did

Seven years ago, I was a self-driving skeptic. Not of the technology. Of all the “experts” promising autonomous vehicles would be everywhere by 2020. You didn’t need to be Nostradamus to know that was ridiculous. All you needed was...
Illustration of a futuristic parking deck turned into a mixed-use space, with AVs driving by

How Autonomous Vehicles Could Help Transform Parking Lots

Researchers say it’s likely that autonomous vehicles (AVs) can help reduce the need for parking lots, opening more room for grass and trees and other elements of nature. It may not seem like it when you’re circling the block...
An illustration of an Argo autonomous vehicle in teal against a backdrop of buildings, a bicyclist, and research papers

7 Big Breakthroughs From Argo Research at CVPR 2022

The 2022 Conference on Computer Vision and Pattern Recognition (CVPR 2022) is nearly here. Thousands of computer scientists, software engineers, and researchers from around the globe will gather in New Orleans to review and discuss their latest work in...

Researchers Predict the Future With Lidar Data

Researchers at Carnegie Mellon University’s Argo AI Center for Autonomous Vehicle Research, a private-public partnership funded by Argo for advancing the autonomous-vehicle (AV) field, say they have come up with a way to use lidar data to visualize not...

Must Reads