The world looks very different at night. Get behind the wheel of a car, and you’ll quickly notice the limitations of human eyesight: In low or no light, we struggle to identify objects. We become reliant on street lights (if there are any). We are blinded by headlights and have a tendency to “overrun” them, because we drive too fast to properly perceive and react to what they reveal. We can be distracted by the glare of our own instrument panels. And then there’s the bleary-eye factor, something familiar to anyone who’s ever commuted home from a long day’s work after sundown, and had trouble identifying dark-colored objects in low-lit areas.
Clearly this is an issue for humans—but how about autonomous vehicles? How are self-driving cars able to “see” at night, and are their powers of detection as strong as they are during the day?
The good news is that autonomous vehicles, such as the ones developed by Argo AI, do not suffer from the same shortcomings. This is for one crucial reason: No matter whether it’s driving during the day or at night, the Argo self-driving system (SDS) effectively “sees” its surroundings the same way, using sensor fusion. This method combines the strengths of several types of sensors to identify the speed, distance, and depth of objects in 360 degrees no matter how dark or light the surroundings. By combining the strengths of lidar, radar, and cameras, Argo uses sensor fusion to ensure that the advantages of each sensor complement the others—an approach that is especially critical for responding to the challenges posed at night.
Putting different sensors to work
What is the most useful sensor technology for driving at night?
Radar sensors emit radio pulses to detect the velocity and range of moving objects, and because radio waves are not impacted by visible light, the technology works as well at noon as it does at midnight. Still, radar can have trouble with identifying an object’s exact shape and precise position.
To reveal those qualities, the Argo SDS utilizes lidar and cameras. The great thing about lidar is, of course, that it is its own light source. In fact, lidar works better when the sun is down, because there is less “solar noise” for it to interpret. The lidar fires rapid laser pulses that bounce off any objects they encounter. The sensor then calculates the precise distance to those objects according to the amount of time the laser beam takes to return, and uses the “point cloud” image—the shape created by the lidar returns—to produce an accurate 3D model of the vehicle’s environment. Currently, the Argo SDS is capable of seeing as far away as 400 meters, regardless of the time of day. Indeed, its proprietary Argo Lidar can also perceive low-reflectivity objects, thanks to its use of Geiger-mode lidar technology, which can detect a single photon, the smallest measurable unit of light. This is particularly useful for black-painted cars due to the minimal light they reflect, which as any human driver will agree, makes them more challenging to detect in the dark, especially when they are parked.
Although cameras excel at object classification and color recognition—both of which help the SDS to discern one object from another—cameras’ limitations come out at night. It’s for this reason that Argo trains its SDS to recognize everything it detects not only in the images from its cameras, but also in the radar scans and the point cloud images created by its lidar sensors. “We pick our cameras to try to get the best low light sensitivity and to get the most out of our headlights. But by design and by nature, a system that uses sensor fusion with lidar, camera, and radar is going to be safer than a camera-only based approach,” says Nicolas Cebron, a senior computer vision manager at Argo AI.
The SDS is also able to classify objects based on their movement, even if a visual classification has not been possible because the object is obscured, such as a person carrying a large trash bin. “If something moves like a pedestrian, the vehicle treats them like a pedestrian even if they weren’t detected that way,” says Argo Chief Technology Officer Dr. Brett Browning.
Detecting flashing lights
Critical to safety of nighttime autonomous driving is the system’s ability to detect moving and stationary fire trucks, ambulances, and police cars, and identify their status. A stationary emergency vehicle may simply be parked and inactive, but it could also be strategically positioned at the side of the road while attending the scene of a major incident.
This is where vehicle light detection comes into play. To identify an active emergency vehicle, an autonomous vehicle must not only identify the vehicle’s shape, size, and features, but also detect its flashing lights. Cebron explains that by using machine learning algorithms which consist of massive amounts of datasets and high-quality annotation, Argo’s SDS cameras are trained to recognize the flashing lights of an emergency vehicle, and combine this data with the 3D point cloud image of the emergency vehicle produced by the Argo Lidar.
Fusing this sensor data completes the picture to ensure the SDS recognizes the presence of a stationary, active emergency vehicle, and safely navigates around it by predicting any actions it might take. But it’s not just stationary emergency vehicles that the Argo SDS can detect—Browning notes that it can also “listen” for the sirens of approaching emergency vehicles, thanks to on-board microphones which add yet another dimension to the vehicle’s sensory capabilities.
The drive for precision
While trusted sensors are already able to detect objects with high precision at night, Argo engineers are continuously working to improve the performance of each one. This is particularly true for Argo’s 12 megapixel cameras whose functionality is constantly refined, such as with the method of zooming in on a “strip” of an image on the horizon to keep that section in high resolution, while downsizing the rest of the image to conserve processing power. Although this technique is not being developed specifically for driving in the dark, it works to enhance the camera’s detectable range, “and that, of course, helps at night,” says Cebron.
It’s the ability of a well-developed self-driving system to detect objects in the dark with strong precision that makes it possible for autonomous vehicles to drive safely at night. By using sensor fusion and spatial context, Argo is designing its self-driving system to accurately detect and recognize critical objects, from emergency vehicles, to pedestrians and cyclists, to construction signs and traffic cones, all while avoiding obvious perception errors—like mistaking the moon for a traffic light.
