As the world looks to shift to cleaner, greener modes of transportation, bicycling can be one of the best, most affordable options. Unfortunately, interactions within traffic can pose a danger. In Miami alone, for example, there were 828 bicycle crashes in 2019, many of which involved cars. So as the Covid-19 pandemic has spurred an uptick in the number of cyclists commuting to work during high traffic times, especially in urban areas, the emphasis on safety has become even more paramount.
New technology aims to address this problem. Argo AI, the self-driving technology company that tests autonomous vehicles on public roads in Miami and seven other cities around the U.S. and Germany, has designed its Self-Driving System (SDS), the set of software and hardware that allows the car to drive on its own, to specifically detect and protect cyclists and other vulnerable road users, including pedestrians, scooter riders, and motorcyclists, with whom it shares the roads.
Here, Andrew Hartnett, the leader of Deep Forecasting at Argo, describes the technological processes going on inside an Argo test vehicle safely passing and stopping for a cyclist in Miami before completing a right turn and avoiding an oncoming vehicle.
The Situation
This scene may appear simple at first, but as Hartnett explains, it’s actually far more complex than it looks.
“The car is estimating the time of interaction and evaluating the implications for the car’s trajectory if it wanted to get ahead or go behind the cyclist,” Hartnett says.
To break it down into its primary parts: 1) the Argo test car comes upon a cyclist traveling in the same direction in a bike lane on the car’s right, 2) the Argo car safely passes the cyclist with ample space between them, 3) the Argo car stops at an intersection for a red light, prepares to turn right on red as is legally permitted in the area but first, 4) waits for the cyclist to pass and enter the intersection, before the car completes its right turn — all without blocking traffic.
If you thought that level of awareness was impressive, the Argo test car follows it up with another safe and smooth driving decision — completing its turn and then slowing and veering to avoid an oncoming vehicle entering its lane after making a U-turn in the middle of the street.
But how does the car do all this, and decide when and where to move to ensure the safety of the cyclist, its occupants, and other vehicles?
Sensing
The process begins with the suite of multiple sensors mounted on the exterior of the Argo test car, which include video cameras, radar, and lidar. They feed data back to the computers inside the vehicle which run the Argo SDS software, including its perception system.
The perception system has the important job of understanding exactly what the car is seeing and, critically, distinguishing the different objects it sees in the world from one another.
In the scenario shown in this video, the Argo SDS must distinguish the moving cyclist from the parked bicycles on the side of the street, as well as separate them all from the concrete median, trees, and buildings surrounding the road.
How does the Argo perception system determine that an object is a bicycle and if it’s parked or if it’s being ridden? It happens through an ongoing process of data collection and image labeling that started long before this or any particular driving interaction and continues in all Argo test cities. From studying numerous images labeled “bicycle,” with the actual bike highlighted in it, the Argo perception system essentially creates for itself a mental “model” of a bicycle that it can refer to, something with wheels and a thin frame, which it can compare to future roadway imagery on the fly.
“For the objects we interact with often, we want to establish the categories that we can recognize,” Hartnett says. “‘Cyclist’ is one of those. This gives the Argo SDS a growing library of labeled roadway imagery to help inform its recognition of objects in the world.
Now take the example in the video: in this case, the car looks at what it’s seeing in real-time as it drives down the street in Miami and compares it to the model of a bicycle it’s created from the labeled imagery Argo cars gathered in the past, also cross-references it with Argo’s on-car, high-resolution local 3D maps that include markings such as bike lanes, where one expects to see cyclists. Using all these tools in an instant, the Argo perception system can reliably and consistently detect bicycles and many other objects 360-degrees around itself.
By using this model, combined with its own frame-to-frame comparison of camera, radar, and lidar data showing where the vehicle is in physical space, the car can tell if a bicycle is moving or stationary.
And, as seen in the video’s data visualization, this happens in roughly the time it takes a human mind to process visual information, but with a much wider field of view — 360 degrees. The Argo perception system successfully identifies all the bicycles around it, both the parked ones and moving one, which is highlighted purple in the visualization.
More than this, the sensor data tells the SDS that the moving bicycle is actually made up of two important separate elements: a rider and a bicycle. The system knows the parked bicycles do not have riders, and therefore, it can move past them as it would other stationary objects.
Planning
Once the SDS recognizes the moving cyclist to its right, it falls upon the motion forecasting and planning systems to determine where the cyclist is likely to move in the near future, and where the Argo car should, in turn, drive to keep the cyclist and the car’s passengers safe.
Before passing the cyclist on the right, Harnett says that the SDS must have observed that the cyclist was following the bike lane closely and wasn’t deviating drastically from it. These few seconds of observation while driving autonomously, combined with the system’s library of previously observed cyclist behaviors, are enough to show the Argo motion forecasting system that the cyclist is riding in such a manner that would allow the car to pass them safely.
“On the prediction side we need to have converged on the fact that the cyclist is following the lane and not crossing into traffic,” Hartnett says.
But, by the same token, Argo’s motion forecasting system is also constantly generating alternate paths that the cyclist may take, assigning those paths probability scores based on the bike’s current and past trajectory. Then, the motion planning system configures the car’s safest possible driving actions in response to all the cyclist’s other potential paths. So, even if the cyclist did dart out into the road, the car is prepared to slow down and ensure everyone continues safely.
The same type of planning calculus takes place inside the SDS not just for this cyclist, but for all the other road actors that the sensors detect — including all the pedestrians walking on the sidewalk and the other vehicles on the road.
In this case, seeing the cyclist is traveling at nearly the same speed as it approaches the red light, and that this speed will put them into the intersection imminently, the SDS understands there is a high probability the cyclist will not stop, and ensures the vehicle waits before making the right turn.
After the autonomous vehicle has made the turn, the motion forecasting and planning systems also enable it to veer out of the way of an oncoming car venturing into its lane.
Acting
It’s not enough for the SDS to understand what the other road actors around it are, what they are doing, likely to do, and what the Argo car should do to preserve the safety of those involved. It must also do all of this while accounting for all local traffic laws, rules, regulations, and norms.
As described in the Argo 2021 Safety Report, “Fundamentally, our SDS makes decisions by selecting driving actions that are verifiably safe given the predictions of other road actors and the rules defined by the traffic control measures in effect.”
In this particular instance, as Hartnett describes the situation, “We want to pass the cyclist, but we also don’t want to infringe on the double yellow line” surrounding the median. So the car rapidly calculates the maneuver it needs to give the cyclist enough space, the lane border enough space, and moves accordingly.
Those actions are initiated by the Argo control system, which connects the car’s computers and software directly into the physical steering, brakes, lights, and other onboard instruments.
The SDS is encoded to stop at all red lights, stop signs, and other traffic signals, as seen in this scene as well. But conducive with the local laws in Miami and signage it sees and that is encoded in its onboard maps, the SDS knows it can turn right on red, so it prepares to do so — but only after the cyclist that is still approaching from behind continues through the intersection and is a safe distance, traveling away from the car.
On top of this, even once the Argo car makes the turn, it sees another vehicle that is jutting slightly into its own lane, and the Argo car veers smoothly to give it space as well. All within just a few seconds, shown in real time in the video.
Sharing the Road Safely, Reliably and Confidently
In order for self-driving cars to be trustworthy, they need to display consistently safe driving behavior and confident decision-making as they move about the real world.
Argo’s SDS is designed around a set of pedestrian and cyclist safety principles with inputs from many cyclist employees and external cycling groups, advocates, and experts. Argo recently partnered with the non-profit cycling advocacy group League of American Bicyclists (LAB) to develop a series of technical guidelines for designing and operating self-driving vehicles around cyclists.
Ultimately, it is this combination of expertise, and the attention to detail in the software systems and local traffic environments, that enable Argo’s SDS to drive smoothly, safely, and confidently around Miami and the other cities in which its self-driving vehicles operate.
