Confocal non-line-of-sight imaging based on the light-cone transform
Matthew O'Toole, David B. Lindell, Gordon Wetzstein
Experimental Setup: The imaging system records the time it takes for laser light to scatter off the wall, reflect off the hidden bunny, and return to the wall. By acquiring these timing measurements for different laser positions on the wall, the 3D geometry of the hidden object can be reconstructed.
A confocal scanning technique solves the reconstruction problem of non-line-of-sight imaging to give fast and high-quality reconstructions of hidden objects.
Abstract
Imaging objects hidden from a camera’s view is a problem of fundamental importance to many fields of research with applications in robotic vision, defense, remote sensing, medical imaging, and autonomous vehicles. Non-line-of-sight (NLOS) imaging at macroscopic scales has been demonstrated by scanning a visible surface with a pulsed laser and time-resolved detector. Whereas light detection and ranging (LIDAR) systems use such measurements to recover the shape of visible objects from direct reflections, NLOS imaging aims at reconstructing the shape and albedo of hidden objects from multiply scattered light. Despite recent advances, NLOS imaging has remained impractical due to the prohibitive memory and processing requirements of existing reconstruction algorithms, and the extremely weak signal of multiply scattered light.
Here we show that confocalizing the scanning procedure provides a means to address these key challenges. Confocal scanning facilitates the derivation of a novel closed-form solution to the NLOS reconstruction problem, which requires computational and memory resources that are orders of magnitude fewer than previous reconstruction methods and recovers hidden objects at unprecedented image resolutions. Confocal scanning also uniquely benefits from a sizeable increase in signal and range when imaging retroreflective objects. We quantify the resolution bounds of NLOS imaging, demonstrate real-time tracking capabilities, and derive efficient algorithms that incorporate image priors and a physically-accurate noise model. Most notably, we demonstrate successful outdoor experiments for NLOS imaging under indirect sunlight.
Stanford News
Overview Video
Visualizations
Measurements and Reconstruction: The captured measurements can be played back as a video, showing light splashing across the wall as it scatters back from the hidden object. Using a mathematical transform and deconvolution algorithm, the hidden object is recorded from these echoes of light.
Outdoor Experiment: Non-line-of-sight imaging is demonstrated outdoors. The imaging system captures measurements in indirect sunlight and robustly reconstructs the hidden “S” shape.
Scenes
Bunny (simulated)
Mannequin
Exit Sign
Letters "SU"
Outdoor Letter "S"
Diffuse "S"
Reconstructions
Media
Matthew O’Toole operating the laser scanning system (credit: Kurt Hickman / Stanford News)
An overview of the hardware setup (credit: Kurt Hickman / Stanford News)
Graduate student David Lindell and postdoctoral researcher Matthew O’Toole work in the lab with assistant professor Gordon Wetzstein (credit: Linda A. Cicero / Stanford News).
Graduate student David Lindell works in the lab with assistant professor Gordon Wetzstein (credit: Linda A. Cicero / Stanford News).
Graduate student David Lindell and postdoctoral researcher Matthew O’Toole calibrating the system (credit: Linda A. Cicero / Stanford News).
Code and Datasets
Acknowledgments
We thank Kai Zang for his expertise and helpful advice related to the SPAD sensor. We also thank Brian A. Wandell, Julie Chang, Isaac Kauvar, and Nitish Padmanaban for reviewing the manuscript. M.O. is supported by the Government of Canada through the Banting Postdoctoral Fellowships program. D.B.L. is supported by a Stanford Graduate Fellowship in Science and Engineering. G.W. is supported by a National Science Foundation CAREER award (IIS 1553333), a Terman Faculty Fellowship, the Center for Automotive Research at Stanford (CARS), and by the KAUST Office of Sponsored Research through the Visual Computing Center CCF grant.
Citation
@article{OToole:2018:ConfocalNLOS,
author = {Matthew O’Toole and David B. Lindell and Gordon Wetzstein},
title = {{Confocal Non-Line-of-Sight Imaging Based on the Light-Cone Transform}},
journal = {Nature},
year = {2018},
}