For experienced drivers, getting from point A to B in a city might seem simple. They survey their surroundings, pick a route, and punch the accelerator.
But this process only appears simple because humans are naturally gifted at quickly inferring how others around them are likely to act, so that they can predict their next moves and react appropriately. A good driver will observe another car approaching a cyclist in the distance and decide to slow down to allow that other car to merge into their lane to avoid colliding with the cyclist sharing its lane.
Veteran drivers are also good at driving “socially” and “defensively.” Socially means they drive in a way that clearly communicates their intentions to everyone else on the road. For example, a driver knows to inch forward at a stop sign to signal that they’re about to proceed. Driving defensively means they anticipate problems before they occur and take actions preemptively. A defensive driver might slow down to increase their following distance if a car in front is acting erratically. This kind of behavior demonstrates situational reasoning that is natural to others on the road. Unusual behavior, on the other hand, can look unnatural—say, a driver maintaining an overly-large following distance even though surrounding traffic is flowing normally, or a driver ignoring apparent risks by tailgating closely behind that erratic car.
The combination of safe, defensive, and socially appropriate driving is what we call “naturalistic driving.” And while developing a safe self-driving vehicle is already a challenge, achieving the level of reasoning and control to achieve naturalistic driving is far more challenging. It takes an army of software developers, machine-learning and computer vision specialists, test engineers and specialists, data analysts, map annotators and object labelers, and more. For self-driving technology company Argo AI, it means designing its self-driving system (SDS) to sense and understand the vehicle’s surroundings, plan future actions, and act on that plan safely and naturalistically.
Here, let’s unpack all the steps that go into making naturalistic driving possible.
Sensing
Before it ventures onto public streets, a self-driving vehicle needs to have an in-depth understanding of its surroundings, and an ability to predict the future actions of all the other actors on the roads. For Argo, this starts with two things: the SDS’s comprehensive 3D map of its environment, and the vehicle’s perception system.
Argo’s 3D maps capture the entire spectrum of static streetscape objects and traffic flow dynamics, equipping Argo self-driving vehicles with “prior knowledge” of a city. The maps don’t just demarcate the intersections, avenues, and side-streets that make up a dense urban environment. They also include traffic signal locations; street signs; local laws and regulations, including speed limits; crosswalks and lane geometries; bike lanes and curbs; buildings and landmarks like statues; even trees.
Using this wealth of contextual knowledge, an Argo self-driving vehicle can then compare what it’s observing in real-time with the detailed 3D model of the physical space. Having a complete understanding of its surroundings enhances the Argo vehicle’s ability to move safely and naturalistically through the world.
Argo’s perception system fuses data from on-board sensors—made up of cameras (stereo and monocular), radars, and lidar (short and long range)—to perceive every actor sharing the road as well as any street markings that may impact driving conditions.
The Argo detection team takes that perception data and uses it to train deep learning models that recognize everything in the vehicle’s environment—including other vehicles, pedestrians, cyclists, animals, road signs, traffic lights, sidewalks, vegetation, and construction zones. Simpler self-driving systems do not recognize all these objects, which limits their ability to interact more naturally with others sharing the road. For example, some systems might only recognize traffic barrels, rather than a construction zone, and therefore might cut in between barrels to get back into a lane that the barrels were attempting to mark as closed.
The detection team supplies image-based information to the Argo tracking team, which fuses it with sensor data to model the movements of the classified objects. This helps the SDS construct a complete, dynamic picture of everything in its environment.
Planning
In a perfect world, everything on the road would follow a set of rules and predictable behaviors. The real world, however, is never this simple. In order to plan how it will get from Point A to B, the Argo SDS has to answer two sets of questions as it drives down the road: What is everyone else going to do next? And how can it use this information to plan a safe, natural motion towards its destination?
To get started, the Argo SDS generates a series of predictions about the future movement of the objects it is tracking. For instance, while a car driving straight will often continue driving straight, the SDS will also predict tens or even hundreds of possible futures, like that car making a right turn at an upcoming intersection, or merging into the lane to its left. Because it is aware of the possibility of a right turn for the other car, the SDS can adjust its speed or lane positioning in anticipation. This is another way that Argo delivers naturalistic driving—subtle actions that help maintain a smooth experience.
While predicting the potential paths of other objects is important, the Argo SDS goes further: it predicts behavior as well. (As an example, see this video, demonstrating how a self-driving vehicle predicts the actions of both pedestrians and traffic.) These ever-changing predictions are based on a variety of inferences about the objects surrounding the vehicle. Predictions consider where an object is facing and the direction in which it’s moving (a car can be facing one direction but moving in reverse). They also take into account scene-level context. For instance, the SDS can detect a crane, a truck, and traffic cones, and infer that it’s in a construction zone. This in turn affects how the SDS estimates the behavior of other actors in the scene. The SDS can also infer that a person crossing the road far away from a crosswalk may be trying to get to a car parked on the other side.
After the Argo SDS takes in its surroundings, it plans its own trajectory for the next few seconds. As it plans, the SDS is searching for the “best” action to take next, considering what it is already doing (driving down the road, say, or waiting at a red light), all the predictions of what other road users are going to do, and how they might interact with the self-driving vehicle.
As the Argo SDS plans its path to its next destination, it needs to maintain safety, comply with the law, and drive naturalistically. If the vehicle approaches a stop sign or red light, for example, it requires itself to stop. It will not change lanes if there is another actor next to it. And while waiting at a stop sign to cross a road with the right of way, the SDS will either wait for traffic to clear, or recognize that it must clear the intersection to avoid forcing a crossing vehicle to slow down unnecessarily. These are all things the SDS must identify and consider in its motion.
The rest of the planning process involves evaluating answers to a slew of additional questions. Is it better to slow down, to shift positions in a lane, or to actually change lanes in response? How far away from a cyclist should the vehicle remain? And is it okay to pass that cyclist, or is it better to slow down and even stop and wait? The more complex the scenario, the more variables there are to weigh. The SDS brings all these variables together to choose an action that is natural, something that both passengers and other road users will understand.
Once the system has sensed its surroundings, predicted future behavior, and planned its optimal trajectory, it’s time to act.
Acting
Watch as an Argo self-driving test vehicle demonstrates naturalistic driving on a recent test-drive in Miami. The SDS detects a cyclist and waits for the cyclist to pass before proceeding with its right turn. It then smoothly slows to allow an oncoming car to move back into the right lane before continuing on its path.
In order to “act,” the Argo SDS is equipped with motion-control capabilities that take action on the trajectory. They direct the engine, braking, and steering systems as the vehicle moves comfortably and safely along a planned path.
The motion-control process considers the force-based dynamics of moving a car, and how they factor into driving naturalistically. To move (or stop) a thing, you have to apply a force. The SDS considers all the real physics that impact how a self-driving vehicle starts, stops, or changes direction on its journey from A to B. For example, cars traveling at 35 mph cannot instantaneously come to a complete stop: they need time and distance for the brakes to slow the car to a stop. The SDS factors everything, including the amount of brake needed and the time it takes to come to a complete stop.
Finally, because every city is unique, and driving norms and laws vary from a place like Miami to Austin to Pittsburgh, Argo’s SDS behavior can be tuned to local driving culture. For instance, Pittsburgh — home of Argo AI’s headquarters — is known for the famous “Pittsburgh left,” during which oncoming traffic gives up the right-of-way and politely lets left-turning vehicles turn against a green. The Argo SDS is trained to anticipate local behavior like the Pittsburgh left in order to yield if appropriate.
In considering any journey of a self-driving vehicle, naturalistic driving and safe driving go hand-in-hand. Gradually inching (instead of darting) into traffic before making a turn, or easing into a new lane on the highway instead of swerving rapidly—these are the subtle, but essential, distinctions that make self-driving vehicles behave naturalistically. If it’s pleasing for riders and recognizable for other road users, then everyone is made safer.
