Hit enter to search or ESC to close

Why Autonomous Vehicles Aren’t Scared of Three-Headed Monsters on Halloween

What happens when a self-driving vehicle comes across something it hasn’t seen before? Something neither man-made, nor of this world? Something…like a three-headed monster?

It’s not a theoretical exercise—especially not on Halloween, when the streets are filled with strange, freakish creatures that are rarely seen on the other 364 days of the year. No matter how otherworldly the encounter, a self-driving system (SDS) still has to detect the beast with multiple heads, deduce that it’s a moving object and not, say, a multi-pronged street sign or an elaborately manicured bush – especially if it is on a pedestrian crossing and has right-of-way – and predict what it will do next. And to do all this, the system must turn to its own secret weapon: open-world detection.

Cutting-edge perception techniques used by the most advanced self-driving systems – such as Argo AI’s – enable autonomous vehicles to detect and respond to known “things” (unitary, countable items, such as pedestrians and other cars) and “stuff” (anything that cannot be counted, such as bushes and hedgerows). But Argo’s system also responds to unknown objects that it has not seen before, which includes moving things that might not be recognized as a person, but move like a person. And that is the function of open-world detection.

A major aspect of training a self-driving system to recognize everything in its environment is labeling objects that it is likely to encounter out on the road, a process which involves the manual annotation of countless images. Everything that a self-driving vehicle might encounter in the real world—bikes, trucks, cars, trees, cats, dogs, mailboxes, traffic lights—will eventually be added to the SDS’s knowledge base. Yet, despite the millions of images used in labeling and training, no perception system will ever recognize everything it sees, and even as the database of labeled images grows, and those unrecognized things become few and far between, the system needs to be able to respond safely when it sees something new. Something like…this:

Dinosaur on a bike

… or this:


Unsurprisingly, open-world detection differs from closed-world detection. Closed-world detection is when the self-driving system detects commonly-encountered objects and recognizes them from a labeling and training process that involves literally millions of images being tagged by meticulous human labelers. Closed-world detection enables the SDS to detect a lamppost; it recognizes a lamppost from its training data—that is, the extensive library of images labeled “lamppost”; and the SDS knows that the lamppost will remain static, so it doesn’t need to predict its movement.

However, a vehicle using an SDS with open-world detection capability is able to carry out complex prediction tasks for uncommon, or never-before-seen objects, including reasoning about that unrecognized object’s likely intentions. The open-world detection system helps the SDS understand that the three-headed monster is human-sized, or that it is behaving in a human-like fashion, on a human-like trajectory – similar to if the vehicle encountered a person holding a large trash bin. It would respond to the three-headed monster as if it was a vulnerable road user, or VRU—the term used for pedestrians, cyclists, micromobility riders, and others road users not protected in the same way that a vehicle has a steel cage—and it would plan safe action in accordance with specifically defined principles. This response includes instructions to the motion planning system to slow the vehicle down or move over, safely putting additional distance between itself and the unidentified road user.

So to you, a mere mortal, that three-headed beast might be the sight that almost stops your heart. But thanks to open-world detection, which ensures that a well-designed SDS behaves in a safe, naturalistic way, even when presented with novel objects, self-driving vehicles aren’t scared of three-headed monsters.

Choose your lane

How Autonomous Vehicles Distinguish Between Bicycles and People Who Ride Them

How Autonomous Vehicles Distinguish Between Bikes and People

When it comes to how autonomous vehicles see the world, humans come first, literally. Autonomous vehicles (AVs), like the kind operated by Pittsburgh-based Argo AI, use Machine Learning to detect and classify the objects in their surroundings, identifying people...
Why The League of American Bicyclists is optimistic about autonomous vehicles

Why a Leading Cycling Advocacy Group Is Optimistic About Autonomous Vehicles

As autonomous vehicle use grows, AV companies and the League of American Bicyclists are collaborating on how to ensure cyclists and motorists can share the roads safely, even if the “motorist” is artificial intelligence software. As part of the...

Self-Driving Is Arriving Right On Time. Just Like Ice Cream Did

Seven years ago, I was a self-driving skeptic. Not of the technology. Of all the “experts” promising autonomous vehicles would be everywhere by 2020. You didn’t need to be Nostradamus to know that was ridiculous. All you needed was...
Illustration of a futuristic parking deck turned into a mixed-use space, with AVs driving by

How Autonomous Vehicles Could Help Transform Parking Lots

Researchers say it’s likely that autonomous vehicles (AVs) can help reduce the need for parking lots, opening more room for grass and trees and other elements of nature. It may not seem like it when you’re circling the block...
An illustration of an Argo autonomous vehicle in teal against a backdrop of buildings, a bicyclist, and research papers

7 Big Breakthroughs From Argo Research at CVPR 2022

The 2022 Conference on Computer Vision and Pattern Recognition (CVPR 2022) is nearly here. Thousands of computer scientists, software engineers, and researchers from around the globe will gather in New Orleans to review and discuss their latest work in...

Researchers Predict the Future With Lidar Data

Researchers at Carnegie Mellon University’s Argo AI Center for Autonomous Vehicle Research, a private-public partnership funded by Argo for advancing the autonomous-vehicle (AV) field, say they have come up with a way to use lidar data to visualize not...

Must Reads