Whenever I’m driving with my mom in the passenger seat, there is a constant monologue of instruction coming out of her: Watch that car ahead of you. Do you see that pedestrian? Go slow, the speed limit just dropped to 35!
Yes, mother, I see it all. I’m just not using my outside voice.
But maybe I should be.
The process of verbalizing observations and upcoming obstacles while behind the wheel is called “commentary driving.” As you likely remember from teenagehood, this technique is often used by driver’s education instructors for helping new drivers to be mindful of their surroundings. At a more extreme level, commentary driving is used by rally car racers to help them navigate the dangerous terrains of off-road races such as the Dakar Rally and Baja 1000.
In the world of autonomous vehicles, however, commentary driving plays a different role, and its effects are helping shape the very future of the technology.
How it works
Self-driving company Argo AI utilizes commentary driving to aid in the safety of testing autonomous vehicles on public roads. Currently, the development phase of self-driving technology involves humans in the front seats. Pairs of Test Specialists, or TSs, accompany Argo self-driving test vehicles, ready to take over if necessary. Argo Training Manager Natalie Ferrer helps teach commentary driving, in which Test Specialists commenting from the left seat, or driver’s position, calmly call out everything from changing traffic conditions to potential obstacles and hazards, as well as passing pedestrians, trees that could obstruct visibility, a pothole, a construction zone, or a stalled car.
(Listen to this audio clip of Argo TSs Tre Davis and Jackworth Smith, on a recent drive in Pittsburgh’s Strip District neighborhood, to hear commentary driving in action.)
Meanwhile a TS in the right (passenger) seat monitors the self-driving system on a laptop, comparing human commentary to the on-screen models. That redundancy helps ensure TSs are aware of everything the vehicle will need to detect, and that they’re prepared to take manual control if needed.
Director of Special Operations and Baja 1000 veteran Alex Roy likens the left seat, right seat relationship to the “symbiotic collaboration” of a driver and navigator on a rally team. When they’re hurtling through an off-road desert course at high speeds, the two have to be perfectly synced. “A race cannot be won in a rally car unless both people understand each other in executing their tasks,” Roy says.
But unlike a rally team, the Test Specialists in a self-driving development vehicle don’t have a marked playbook. “While they know the terrain and geography, there is an enormous amount of observation required to monitor the road ahead.”
Retraining the brain
To trainers like Ferrer, commentary driving isn’t just how partners stay connected — it’s a way of improving situational awareness by reteaching their brains how to drive. It also adds another layer of safety by keeping the TSs alert. The technique, which is taught to all Test Specialists over a four-week training period, “makes you slow your brain and the processes down, forcing you to be more vigilant, more safe, more in the present moment,” she says. Take, for example, calling out a stop sign—stating the presence of a stop sign up ahead prompts the driver to evaluate the next steps and brake early enough for a smooth stop at the proper location.
Argo AI trainers use the “SEE” method, a process borrowed from motorcycle-safety courses, which stands for Search, Evaluate and Execute. During the Searching phase, Test Specialists use another acronym, this one invented by Argo, so they know what they’re calling out: CLEAR, or Crosswalks, Lights and Lane Laws, Environments, Actors, and Routes. Busy city streets often have dozens of people and situations that might fit into these categories. So which of these takes precedence?
In essence, a good TS is vigilant, prioritizes quickly, trusts the vehicle when appropriate and takes over when they deem appropriate for the situation. That starts in the classroom with candidates watching situational videos and commenting to instructors, then hearing how the instructor might have handled the same scene. If there are differences, the trainee and instructor discuss why. For example, a trainee might call out the existence of a parking lot but miss a traffic-light change from green to yellow. In that instance, Ferrer might say: “That light just changed color, and that takes priority over saying that the parking lot to your right is empty.”
High standards
But why does all this matter? Because accurate commentary helps ensure the self-driving system behaves appropriately. Every manual takeover, or disengagement, is logged into the vehicle’s trip data, which is then reviewed by an Argo data analysis team. This step triggers a rigorous software-updating process that, broadly speaking, involves virtual testing, reprogramming the system, and plugging revised code back into the testing cycle. This is what Argo AI calls a “continuous cycle of testing,” as every trip’s log provides an opportunity to improve the software.
So, while one purpose of commentary is to maintain TS mindfulness, another, equally important purpose is to verify, on a second-to-second basis, that the self-driving system is constantly improving.
In this way, commentary driving is so much more than just acknowledging every pedestrian or red light that a TS (or my mother) sees. It’s about seeing the world in all its unpredictability and randomness, and responding with confidence.
