Hit enter to search or ESC to close

Watch and Learn: An Autonomous Vehicle and a Pickup Truck Cross Paths In Pittsburgh

A colorful illustration showing a screenshot from a video from the perspective of an Argo autonomous vehicle

In order for self-driving cars to integrate smoothly with other people and vehicles on the roads, they must do more than drive intelligently and safely: they must also drive naturalistically, like an experienced local driver.

In this Watch and Learn article, Constantin Savtchenko, a Senior Staff Engineer at self-driving technology company Argo AI, takes us through a video of an Argo autonomous vehicle (AV) exhibiting naturalistic driving qualities on a narrow street in Pittsburgh, Pennsylvania during public road testing.

While deceptively simple at first, the video shows off a complex interaction that requires sophisticated understanding of local and social roadway behavior by an Argo AV.

The video shows that the AV is “working exactly as designed,” Savtchenko says, the culmination of dozens of teams of engineers working round-the-clock on the the Argo Self-Driving System (SDS), the collection of hardware and software that allows the car to drive and make decisions on its own.

The Situation

This video shows the AV driving autonomously down one of Pittsburgh’s many narrow two-lane streets, made even narrower in this case by parked vehicles along both sides.

The AV approaches a T-shaped intersection with a stop sign and comes to a complete stop in preparation to turn left onto the perpendicular cross-street.

However, before it can begin to safely proceed, a pickup truck pulls up from the left side of the cross-street and begins to turn right into the AV’s path. Factoring in the parked cars and the narrowness of the street, there’s not enough space for both vehicles to continue moving side-by-side.

Clearly, they need to take turns to continue. But how does the AV know when to take its turn and how to move safely through this situation?


First, the AV must accurately detect what is going on around it, using a suite of multiple sensors mounted on its exterior, including video cameras, radar, microphones and lidar.

These sensors all feed different types of data into the SDS, which uses a process called  “sensor fusion,” to combine all the data from its various sensors together, creating a high-resolution, highly accurate, constantly updated 3D image of what’s happening 360 degrees around the AV, according to Savtchenko.

The perception system of the SDS then categorizes what the AV is seeing around into different groups: pedestrian, other vehicle, bicycle, stop sign, and many more. By understanding the difference between these groups and object types, the SDS can then adjust its expectations of what these objects are likely to do, and how it should drive to maintain their safety.

“Many moving parts come together in unison to estimate what’s going on around the AV,” Savtchenko says. “From actions that sound simple — such as taking camera images or point clouds — to actions that seems complex — such as state of the art algorithms to categorize all the objects in the image, our system does it all in milliseconds, pushing the bounds of what is possible on today’s hardware while maintaining stability and consistency.”


The AV needs to decide when and how to move in relation to the oncoming pickup truck.

This decision falls to the motion planning and control (MPC) part of the SDS, software that takes the data from the sensors and perception system indicating how fast the truck is moving and where, and uses these values to predict a number of possible and likely paths that the truck — and any other moving objects that may appear — will take.

“The motion planning system continuously takes the forecasts of other actors and vehicles generated by motion forecasting and generates safe plans for the AV using the many possible futures,” Savtchenko says.

The MPC performs these complex calculations not just once, but several times per second, continuously recalculating as the scenario around the AV changes.

Each time it goes through a planning cycle, the MPC needs to ensure its actions are consistent with local driving rules.

In this case, the applicable rules  are Pennsylvania traffic laws that should be familiar to drivers in other states as well: vehicles that have a stop sign must stop, while those that have no signs may proceed through an intersection.

The AV stops as required, but surprisingly the truck also stops — which is not required. The AV can respond in one of two ways:

  • The SDS can estimate a causal reason for why the truck is not proceeding, such as they don’t want to hit the AV and the AV is blocking their path
  • The SDS can take the situation at face value that the truck has stopped, regardless of reason.

In this case, with no other traffic in sight, the AV can assess with high confidence the truck has stopped because it doesn’t want to infringe on the AV’s path.

The AV concludes the safest move is to move out of the truck’s way, so the MPC generates a safe and lawful path for the AV to move around it.


After the MPC calculates the AV’s path with onboard computers, it sends the instructions to the car’s controls to carry them out — i.e. steering wheel, accelerator, brakes, turn signal indicators.

“Once a plan is proposed and evaluated against possible outcomes, the motion planning system sends it to motion control for execution,” Savtchenko explains. “From here motion control utilizes a complex system of equations to interact with the vehicle (think gas,brake pedals, and steering wheel) to naturalistically execute that plan.”

‘Normal and unremarkable’

Naturalistic driving ensures that the Argo SDS is able to react flexibly and appropriately, even in unclear situations with other human drivers.

“We engineer different modules for many different parts of the SDS,” Savtchenko says, referring to programs that are linked together. “And then when they all work together in unison, the result is a car that drives safely and comfortably according to the rules of the road.”

Choose your lane

How Autonomous Vehicles Distinguish Between Bicycles and People Who Ride Them

How Autonomous Vehicles Distinguish Between Bikes and People

When it comes to how autonomous vehicles see the world, humans come first, literally. Autonomous vehicles (AVs), like the kind operated by Pittsburgh-based Argo AI, use Machine Learning to detect and classify the objects in their surroundings, identifying people...
Why The League of American Bicyclists is optimistic about autonomous vehicles

Why a Leading Cycling Advocacy Group Is Optimistic About Autonomous Vehicles

As autonomous vehicle use grows, AV companies and the League of American Bicyclists are collaborating on how to ensure cyclists and motorists can share the roads safely, even if the “motorist” is artificial intelligence software. As part of the...

Self-Driving Is Arriving Right On Time. Just Like Ice Cream Did

Seven years ago, I was a self-driving skeptic. Not of the technology. Of all the “experts” promising autonomous vehicles would be everywhere by 2020. You didn’t need to be Nostradamus to know that was ridiculous. All you needed was...
Illustration of a futuristic parking deck turned into a mixed-use space, with AVs driving by

How Autonomous Vehicles Could Help Transform Parking Lots

Researchers say it’s likely that autonomous vehicles (AVs) can help reduce the need for parking lots, opening more room for grass and trees and other elements of nature. It may not seem like it when you’re circling the block...
An illustration of an Argo autonomous vehicle in teal against a backdrop of buildings, a bicyclist, and research papers

7 Big Breakthroughs From Argo Research at CVPR 2022

The 2022 Conference on Computer Vision and Pattern Recognition (CVPR 2022) is nearly here. Thousands of computer scientists, software engineers, and researchers from around the globe will gather in New Orleans to review and discuss their latest work in...

Researchers Predict the Future With Lidar Data

Researchers at Carnegie Mellon University’s Argo AI Center for Autonomous Vehicle Research, a private-public partnership funded by Argo for advancing the autonomous-vehicle (AV) field, say they have come up with a way to use lidar data to visualize not...

Must Reads