Hit enter to search or ESC to close

Why Autonomous Vehicles Aren’t Scared of Three-Headed Monsters on Halloween

What happens when a self-driving vehicle comes across something it hasn’t seen before? Something neither man-made, nor of this world? Something…like a three-headed monster?

It’s not a theoretical exercise—especially not on Halloween, when the streets are filled with strange, freakish creatures that are rarely seen on the other 364 days of the year. No matter how otherworldly the encounter, a self-driving system (SDS) still has to detect the beast with multiple heads, deduce that it’s a moving object and not, say, a multi-pronged street sign or an elaborately manicured bush – especially if it is on a pedestrian crossing and has right-of-way – and predict what it will do next. And to do all this, the system must turn to its own secret weapon: open-world detection.

Cutting-edge perception techniques used by the most advanced self-driving systems – such as Argo AI’s – enable autonomous vehicles to detect and respond to known “things” (unitary, countable items, such as pedestrians and other cars) and “stuff” (anything that cannot be counted, such as bushes and hedgerows). But Argo’s system also responds to unknown objects that it has not seen before, which includes moving things that might not be recognized as a person, but move like a person. And that is the function of open-world detection.

A major aspect of training a self-driving system to recognize everything in its environment is labeling objects that it is likely to encounter out on the road, a process which involves the manual annotation of countless images. Everything that a self-driving vehicle might encounter in the real world—bikes, trucks, cars, trees, cats, dogs, mailboxes, traffic lights—will eventually be added to the SDS’s knowledge base. Yet, despite the millions of images used in labeling and training, no perception system will ever recognize everything it sees, and even as the database of labeled images grows, and those unrecognized things become few and far between, the system needs to be able to respond safely when it sees something new. Something like…this:

Dinosaur on a bike

… or this:


Unsurprisingly, open-world detection differs from closed-world detection. Closed-world detection is when the self-driving system detects commonly-encountered objects and recognizes them from a labeling and training process that involves literally millions of images being tagged by meticulous human labelers. Closed-world detection enables the SDS to detect a lamppost; it recognizes a lamppost from its training data—that is, the extensive library of images labeled “lamppost”; and the SDS knows that the lamppost will remain static, so it doesn’t need to predict its movement.

However, a vehicle using an SDS with open-world detection capability is able to carry out complex prediction tasks for uncommon, or never-before-seen objects, including reasoning about that unrecognized object’s likely intentions. The open-world detection system helps the SDS understand that the three-headed monster is human-sized, or that it is behaving in a human-like fashion, on a human-like trajectory – similar to if the vehicle encountered a person holding a large trash bin. It would respond to the three-headed monster as if it was a vulnerable road user, or VRU—the term used for pedestrians, cyclists, micromobility riders, and others road users not protected in the same way that a vehicle has a steel cage—and it would plan safe action in accordance with specifically defined principles. This response includes instructions to the motion planning system to slow the vehicle down or move over, safely putting additional distance between itself and the unidentified road user.

So to you, a mere mortal, that three-headed beast might be the sight that almost stops your heart. But thanks to open-world detection, which ensures that a well-designed SDS behaves in a safe, naturalistic way, even when presented with novel objects, self-driving vehicles aren’t scared of three-headed monsters.

Must Reads