“They came out of nowhere!” You’ve probably heard this phrase uttered while riding in a car (maybe you’ve even said it yourself) if another vehicle, pedestrian, or bicyclist appears suddenly in front of your vehicle without warning.
Safe drivers will react by slowing, stopping, or moving their car appropriately to avoid contact with the fellow road user who took them by surprise, using their senses, reflexes, and experience/driving education to help guide their actions.
But what about a self-driving car? How does it respond to this common real-world driving scenario? And what processes or calculations go into its reaction?
Watch how Argo AI’s self-driving test vehicle safely navigates an unpredictable scenario in Miami, Florida and read about the technological process behind it with explanation from Peter Carr, Argo VP of Autonomy and Software Engineering
The Situation
As shown in the video, the Argo AI test car is driving in autonomous mode down a two-way, two-lane street. It comes upon an intersection with a four-way stop, brakes accordingly to a standstill, and begins creeping forward again, slowly to ensure perpendicular traffic is clear — before a cyclist quickly emerges from behind a prominent building on the car’s right side and rides down the sidewalk out in front of it. The Argo car reacts safely by braking, coming to a prompt yet gentle halt, allowing the cyclist to pass from right to left with plenty of space between them and the Argo car, all while keeping the car’s occupants comfortable. Then, the Argo car continues on its way.
What is going on inside the car, in Argo’s self-driving system (SDS) — the collection of hardware sensors, computers, and software that enable autonomous driving — that enables it to respond so effectively and safely in this type of fast-moving situation? Carr explains the basic elements below.
Sensing
First, the Argo car instantly detects the bicyclist traveling quickly on the sidewalk, and classifies them as a cyclist, rather than another vehicle, pedestrian, animal, or part of the environment, such as a tree branch or piece of roadway infrastructure. Cyclists are deemed “vulnerable road users,” (VRUs) because they lack the protection offered by a car, truck, or other larger vehicle body. Pedestrians, motorcyclists, and scooter riders are also VRUs.
The detection is done first by using sensor fusion, the process of combining data streams from multiple sensors — in this case, namely, cameras and lidar mounted outside of the car — and then running them through Argo’s perception system, which compares it to a database of tagged imagery from previous test rides, to understand rapidly, in fractions of a second, that this is a 3D object, that the object is actually two things, a person and a bicycle, and that it is moving at a specific velocity and direction.
“The camera sees that there is a bicycle and rider on the sidewalk. Similarly, the lidar senses a point cloud and agrees, ‘this must be a bicyclist,’” Carr says. “The SDS also determines the bicyclist’s speed and direction of travel. A person riding on or walking the bike creates a difference in the software’s response.”
Planning
It’s not enough to simply see and understand that a bicycle is moving in front of the car. A safe self-driving system must also be able to anticipate where the bicycle is likeliest to move next, as well as additional paths it could take and the probability of it taking those paths, and finally, how the car itself should respond accordingly in each of the ways the cyclist may move.
In the Argo SDS, this series of tasks is the responsibility of the motion forecasting software, a collection of sophisticated, fast algorithms that calculates, based on what type of vehicle or objects the SDS has detected, their current speed and direction, and where they could and are most likely to move several seconds into the future. And, depending on where the cyclist in this case actually does move, the motion forecasting software updates its predictions multiple times per second to account for numerous potential paths of every object.
Argo’s SDS is specifically attuned to bicyclists by anticipating them riding on the road as a vehicle would, or across sidewalks where a pedestrian would walk, as in this example.
“The Argo SDS is built to forecast transitions between road and sidewalk,” Carr says. “Even if it’s not completely obvious whether the cyclist has seen the autonomous vehicle (AV) or not, or is planning to stop or not, the Argo SDS has forecasted both of those futures — and many others.”
Acting
Once the Argo SDS has identified what it sees as a cyclist, and forecasted their possible paths of motion and which is most likely, it rapidly determines what, if any, driving actions should be completed.
In this particular case, the Argo SDS brakes and waits for the cyclist to ride through the intersection. But, as is the case with human drivers, the Argo SDS must also answer the question of how hard to apply the brakes?
“It’s a combination of safety and ride comfort — can it come to a stop without slamming on the brakes?” Carr explains. “If so, the car does that. You want to make sure the SDS does exactly the amount of braking necessary to be safe and no more. If the situation was different and the cyclist rode faster or closer, the car would undertake harder braking. But smoother braking is preferred whenever it’s safe to give vehicles behind the AV a chance to stop as well.”
Furthermore, the Argo SDS also waits to resume driving until the bicyclist passes fully through the intersection and is far enough away that if they were to veer left or right, they would not encroach on the forward path of the car.
Keeping Cyclists and Other Road Users Safe
As this video and Carr’s explanations illustrate, Argo keeps cyclists and other road users safe through a thoughtful, comprehensive, and holistic approach — taking into account the cyclist and the car, as well as the entire scene around them. It does not rely on any particular idealistic cyclist behavior or set of behaviors, but rather on the real-world range of potential behaviors any cyclist can make along their ride.
Argo’s SDS has been designed with contributions from cyclists on Argo’s staff and other valued advisors, like the League of American Bicyclists. With the help of this input, the SDS follows a set of principles for pedestrian and cyclist safety that prioritizes watching out for them and other vulnerable road users.
