Hit enter to search or ESC to close

Two Years Into Argo AI and Carnegie Mellon’s Quest to Drive What’s Next In Autonomous Vehicles

A bleeding edge technology like autonomous driving requires cutting-edge scientific research to support it. And that research can never stand still–it must keep up with the industry as it changes, and pave the way for greater breakthroughs to come. 

That was the impetus behind the establishment of the CMU Argo AI Center for Autonomous Vehicle Research in 2019. The five-year, $15 million partnership between Carnegie Mellon University and the Pittsburgh-based self-driving technology company, Argo AI, promised to tie industry and academia closely together, resulting in breakthroughs in the fields supporting autonomy. The partnership is designed to allow Argo to provide CMU students with access to data, infrastructure, and platforms, and for research, software, and datasets to remain open for use by the research community at large. 

Two years in, the partnership has yielded more than a dozen published projects and papers. This work has led to new findings in areas as disparate as smart sensor fusion, 3D scene understanding, urban scene simulation, map-based perception, imitation and reinforcement learning, behavioral prediction, and robust validation of software. 

Under the leadership of Deva Ramanan, associate professor in the Robotics Institute and principal scientist at Argo AI, such scientific discoveries continue apace. Below, Ramanan highlights just three recent standout projects that are moving the industry forward.


The Abstract

Compressed 3D Map for Monocular Camera Registration

Termed Hypermap, this lays out a new approach to determining an autonomous vehicle’s location on a 3D map of a city. It puts more machine learning into the pipeline of building a map, making online (on-vehicle) localization more accurate and efficient.

The Authors

CMU PhD student Ming-Fang Chang, assistant professor Joshua Mangelson, and associate professors Michael Kaess and Simon Lucey.

The Problem 

Autonomous vehicles compare what they perceive in their surroundings using lidar and other sensors to the 3D maps stored in the vehicle’s memory. However, current approaches to autonomous vehicle localization require a huge amount of on-car compute power, increasing the energy consumption and memory needed to accurately determine a vehicle’s location. 

The Breakthrough 

When determining a vehicle’s location, instead of matching only the lidar point clouds observed by the autonomous vehicle to a stored 3D map, Hypermap explores alternative modalities for localization and map building, such as using the vehicle’s RGB (red, green, blue) cameras. This allows for the creation of maps that include recorded road features, like lane lines, physical obstacles, and traffic signs. The engineers building these maps can then deploy neural networks to identify and define those features, which trains the map to be more detailed and accurate. Such networks reduce the compute power needed to identify vehicle location in real-time, while also increasing the accuracy and efficiency of localization. 


The Abstract

Behavior Planning at Urban Intersections Through Hierarchical Reinforcement Learning

A new approach to autonomous-behavior planning that uses reinforcement learning models for safer, more natural driving in complex environments. 

The Authors

CMU research assistant Zhiqian Qiao, research professor Jeff Schneider, and principal scientist John M. Dolan. 

The Problem 

Effective behavior planning is crucial to ensure the safety of an autonomous vehicle, but it is difficult to create sufficiently general heuristic rules (when you see x, do y), especially for particularly challenging scenarios, like turning left through oncoming traffic.

The Breakthrough 

This project proposes a new behavior-planning approach for autonomous vehicles that relies on a hybrid approach of both heuristic rules and traditional reinforcement learning—training machine learning models to make a sequence of decisions by using trial and error, rewarding the system for the “right” response and penalizing errors. The resulting hybrid system has the benefit of performing more safely and naturally than solely heuristic-rule-based methods, and requires less data to refine than traditional reinforcement learning methods. 


The Abstract

Safe Local Motion Planning with Self-Supervised Freespace Forecasting

A new method that uses “freespace,” or regions that are safe for an autonomous vehicle to maneuver into, to improve the safety of an autonomous vehicle’s motion planning. 

The Authors

CMU PhD candidate, Peiyun Hu, graduate research assistant, Aaron Huang, principal scientist, John Dolan, assistant professor, David Held, and Deva Ramanan

The Problem 

Autonomous systems typically perceive their surroundings by detecting, tracking, and then forecasting the locations of a known library of objects. These perception outputs are fed into the vehicle’s motion planning system to generate safe, naturalistic trajectories of motion. However, such motion planning pipelines can be cumbersome to train since they require lots of annotated data.

The Breakthrough 

Instead of detecting and forecasting the locations of every object in its surrounding, this project explores the provocative idea that an autonomous vehicle can forecast freespace—the space between a vehicle and an object registered by lidar—to improve safe motion planning. Since freespace is only a measure of the space between the autonomous vehicle and an object it observes, freespace outputs can be learned in a “self-supervised manner,” or without manual annotation of every object in its surroundings. This allows the system to train on considerably more data, increasing its accuracy. In early simulated trials, this method has been shown to significantly reduce risk mitigation.


Growing Together

Looking forward, Argo and the CMU center continue to grow on parallel tracks. As Argo makes strides toward launching a commercial self-driving service, CMU researchers reap the benefits, developing more advanced projects that in turn help Argo’s engineers improve their self-driving system. One recent example of this mutual growth comes in the form of Argoverse 2.0, Argo’s freshly released public-facing datasets that provide new map and sensor data from six cities, and a correspondingly larger playground for CMU’s PhD’s and professors to train their projects. CMU researchers are also leveraging Argo’s new longer range Lidar technology to refine projects like Freespace Forecasting, and to develop new projects entirely.

Choose your lane

How Autonomous Vehicles Distinguish Between Bicycles and People Who Ride Them

How Autonomous Vehicles Distinguish Between Bikes and People

When it comes to how autonomous vehicles see the world, humans come first, literally. Autonomous vehicles (AVs), like the kind operated by Pittsburgh-based Argo AI, use Machine Learning to detect and classify the objects in their surroundings, identifying people...
Why The League of American Bicyclists is optimistic about autonomous vehicles

Why a Leading Cycling Advocacy Group Is Optimistic About Autonomous Vehicles

As autonomous vehicle use grows, AV companies and the League of American Bicyclists are collaborating on how to ensure cyclists and motorists can share the roads safely, even if the “motorist” is artificial intelligence software. As part of the...

Self-Driving Is Arriving Right On Time. Just Like Ice Cream Did

Seven years ago, I was a self-driving skeptic. Not of the technology. Of all the “experts” promising autonomous vehicles would be everywhere by 2020. You didn’t need to be Nostradamus to know that was ridiculous. All you needed was...
Illustration of a futuristic parking deck turned into a mixed-use space, with AVs driving by

How Autonomous Vehicles Could Help Transform Parking Lots

Researchers say it’s likely that autonomous vehicles (AVs) can help reduce the need for parking lots, opening more room for grass and trees and other elements of nature. It may not seem like it when you’re circling the block...
An illustration of an Argo autonomous vehicle in teal against a backdrop of buildings, a bicyclist, and research papers

7 Big Breakthroughs From Argo Research at CVPR 2022

The 2022 Conference on Computer Vision and Pattern Recognition (CVPR 2022) is nearly here. Thousands of computer scientists, software engineers, and researchers from around the globe will gather in New Orleans to review and discuss their latest work in...

Researchers Predict the Future With Lidar Data

Researchers at Carnegie Mellon University’s Argo AI Center for Autonomous Vehicle Research, a private-public partnership funded by Argo for advancing the autonomous-vehicle (AV) field, say they have come up with a way to use lidar data to visualize not...

Must Reads