In order for self-driving cars to integrate smoothly with other people and vehicles on the roads, they must do more than drive intelligently and safely: they must also drive naturalistically, like an experienced local driver.
In this Watch and Learn article, Constantin Savtchenko, a Senior Staff Engineer at self-driving technology company Argo AI, takes us through a video of an Argo autonomous vehicle (AV) exhibiting naturalistic driving qualities on a narrow street in Pittsburgh, Pennsylvania during public road testing.
While deceptively simple at first, the video shows off a complex interaction that requires sophisticated understanding of local and social roadway behavior by an Argo AV.
The video shows that the AV is “working exactly as designed,” Savtchenko says, the culmination of dozens of teams of engineers working round-the-clock on the the Argo Self-Driving System (SDS), the collection of hardware and software that allows the car to drive and make decisions on its own.
The Situation
This video shows the AV driving autonomously down one of Pittsburgh’s many narrow two-lane streets, made even narrower in this case by parked vehicles along both sides.
The AV approaches a T-shaped intersection with a stop sign and comes to a complete stop in preparation to turn left onto the perpendicular cross-street.
However, before it can begin to safely proceed, a pickup truck pulls up from the left side of the cross-street and begins to turn right into the AV’s path. Factoring in the parked cars and the narrowness of the street, there’s not enough space for both vehicles to continue moving side-by-side.
Clearly, they need to take turns to continue. But how does the AV know when to take its turn and how to move safely through this situation?
Sense
First, the AV must accurately detect what is going on around it, using a suite of multiple sensors mounted on its exterior, including video cameras, radar, microphones and lidar.
These sensors all feed different types of data into the SDS, which uses a process called “sensor fusion,” to combine all the data from its various sensors together, creating a high-resolution, highly accurate, constantly updated 3D image of what’s happening 360 degrees around the AV, according to Savtchenko.
The perception system of the SDS then categorizes what the AV is seeing around into different groups: pedestrian, other vehicle, bicycle, stop sign, and many more. By understanding the difference between these groups and object types, the SDS can then adjust its expectations of what these objects are likely to do, and how it should drive to maintain their safety.
“Many moving parts come together in unison to estimate what’s going on around the AV,” Savtchenko says. “From actions that sound simple — such as taking camera images or point clouds — to actions that seems complex — such as state of the art algorithms to categorize all the objects in the image, our system does it all in milliseconds, pushing the bounds of what is possible on today’s hardware while maintaining stability and consistency.”
Plan
The AV needs to decide when and how to move in relation to the oncoming pickup truck.
This decision falls to the motion planning and control (MPC) part of the SDS, software that takes the data from the sensors and perception system indicating how fast the truck is moving and where, and uses these values to predict a number of possible and likely paths that the truck — and any other moving objects that may appear — will take.
“The motion planning system continuously takes the forecasts of other actors and vehicles generated by motion forecasting and generates safe plans for the AV using the many possible futures,” Savtchenko says.
The MPC performs these complex calculations not just once, but several times per second, continuously recalculating as the scenario around the AV changes.
Each time it goes through a planning cycle, the MPC needs to ensure its actions are consistent with local driving rules.
In this case, the applicable rules are Pennsylvania traffic laws that should be familiar to drivers in other states as well: vehicles that have a stop sign must stop, while those that have no signs may proceed through an intersection.
The AV stops as required, but surprisingly the truck also stops — which is not required. The AV can respond in one of two ways:
- The SDS can estimate a causal reason for why the truck is not proceeding, such as they don’t want to hit the AV and the AV is blocking their path
- The SDS can take the situation at face value that the truck has stopped, regardless of reason.
In this case, with no other traffic in sight, the AV can assess with high confidence the truck has stopped because it doesn’t want to infringe on the AV’s path.
The AV concludes the safest move is to move out of the truck’s way, so the MPC generates a safe and lawful path for the AV to move around it.
Act
After the MPC calculates the AV’s path with onboard computers, it sends the instructions to the car’s controls to carry them out — i.e. steering wheel, accelerator, brakes, turn signal indicators.
“Once a plan is proposed and evaluated against possible outcomes, the motion planning system sends it to motion control for execution,” Savtchenko explains. “From here motion control utilizes a complex system of equations to interact with the vehicle (think gas,brake pedals, and steering wheel) to naturalistically execute that plan.”
‘Normal and unremarkable’
Naturalistic driving ensures that the Argo SDS is able to react flexibly and appropriately, even in unclear situations with other human drivers.
“We engineer different modules for many different parts of the SDS,” Savtchenko says, referring to programs that are linked together. “And then when they all work together in unison, the result is a car that drives safely and comfortably according to the rules of the road.”
