Two Years Into Argo AI and Carnegie Mellon’s Quest to Drive What’s Next In Autonomous Vehicles

A bleeding edge technology like autonomous driving requires cutting-edge scientific research to support it. And that research can never stand still–it must keep up with the industry as it changes, and pave the way for greater breakthroughs to come. 

That was the impetus behind the establishment of the CMU Argo AI Center for Autonomous Vehicle Research in 2019. The five-year, $15 million partnership between Carnegie Mellon University and the Pittsburgh-based self-driving technology company, Argo AI, promised to tie industry and academia closely together, resulting in breakthroughs in the fields supporting autonomy. The partnership is designed to allow Argo to provide CMU students with access to data, infrastructure, and platforms, and for research, software, and datasets to remain open for use by the research community at large. 

Two years in, the partnership has yielded more than a dozen published projects and papers. This work has led to new findings in areas as disparate as smart sensor fusion, 3D scene understanding, urban scene simulation, map-based perception, imitation and reinforcement learning, behavioral prediction, and robust validation of software. 

Under the leadership of Deva Ramanan, associate professor in the Robotics Institute and principal scientist at Argo AI, such scientific discoveries continue apace. Below, Ramanan highlights just three recent standout projects that are moving the industry forward.

The Abstract

Compressed 3D Map for Monocular Camera Registration

Termed Hypermap, this lays out a new approach to determining an autonomous vehicle’s location on a 3D map of a city. It puts more machine learning into the pipeline of building a map, making online (on-vehicle) localization more accurate and efficient.

The Authors

CMU PhD student Ming-Fang Chang, assistant professor Joshua Mangelson, and associate professors Michael Kaess and Simon Lucey.

The Problem 

Autonomous vehicles compare what they perceive in their surroundings using lidar and other sensors to the 3D maps stored in the vehicle’s memory. However, current approaches to autonomous vehicle localization require a huge amount of on-car compute power, increasing the energy consumption and memory needed to accurately determine a vehicle’s location. 

The Breakthrough 

When determining a vehicle’s location, instead of matching only the lidar point clouds observed by the autonomous vehicle to a stored 3D map, Hypermap explores alternative modalities for localization and map building, such as using the vehicle’s RGB (red, green, blue) cameras. This allows for the creation of maps that include recorded road features, like lane lines, physical obstacles, and traffic signs. The engineers building these maps can then deploy neural networks to identify and define those features, which trains the map to be more detailed and accurate. Such networks reduce the compute power needed to identify vehicle location in real-time, while also increasing the accuracy and efficiency of localization. 

The Abstract

Behavior Planning at Urban Intersections Through Hierarchical Reinforcement Learning

A new approach to autonomous-behavior planning that uses reinforcement learning models for safer, more natural driving in complex environments. 

The Authors

CMU research assistant Zhiqian Qiao, research professor Jeff Schneider, and principal scientist John M. Dolan. 

The Problem 

Effective behavior planning is crucial to ensure the safety of an autonomous vehicle, but it is difficult to create sufficiently general heuristic rules (when you see x, do y), especially for particularly challenging scenarios, like turning left through oncoming traffic.

The Breakthrough 

This project proposes a new behavior-planning approach for autonomous vehicles that relies on a hybrid approach of both heuristic rules and traditional reinforcement learning—training machine learning models to make a sequence of decisions by using trial and error, rewarding the system for the “right” response and penalizing errors. The resulting hybrid system has the benefit of performing more safely and naturally than solely heuristic-rule-based methods, and requires less data to refine than traditional reinforcement learning methods. 

The Abstract

Safe Local Motion Planning with Self-Supervised Freespace Forecasting

A new method that uses “freespace,” or regions that are safe for an autonomous vehicle to maneuver into, to improve the safety of an autonomous vehicle’s motion planning. 

The Authors

CMU PhD candidate, Peiyun Hu, graduate research assistant, Aaron Huang, principal scientist, John Dolan, assistant professor, David Held, and Deva Ramanan

The Problem 

Autonomous systems typically perceive their surroundings by detecting, tracking, and then forecasting the locations of a known library of objects. These perception outputs are fed into the vehicle’s motion planning system to generate safe, naturalistic trajectories of motion. However, such motion planning pipelines can be cumbersome to train since they require lots of annotated data.

The Breakthrough 

Instead of detecting and forecasting the locations of every object in its surrounding, this project explores the provocative idea that an autonomous vehicle can forecast freespace—the space between a vehicle and an object registered by lidar—to improve safe motion planning. Since freespace is only a measure of the space between the autonomous vehicle and an object it observes, freespace outputs can be learned in a “self-supervised manner,” or without manual annotation of every object in its surroundings. This allows the system to train on considerably more data, increasing its accuracy. In early simulated trials, this method has been shown to significantly reduce risk mitigation.

Growing Together

Looking forward, Argo and the CMU center continue to grow on parallel tracks. As Argo makes strides toward launching a commercial self-driving service, CMU researchers reap the benefits, developing more advanced projects that in turn help Argo’s engineers improve their self-driving system. One recent example of this mutual growth comes in the form of Argoverse 2.0, Argo’s freshly released public-facing datasets that provide new map and sensor data from six cities, and a correspondingly larger playground for CMU’s PhD’s and professors to train their projects. CMU researchers are also leveraging Argo’s new longer range Lidar technology to refine projects like Freespace Forecasting, and to develop new projects entirely.