We introduce a wave-based image formation model for the problem of non-line-of-sight (NLOS) imaging. Inspired by inverse methods used in seismology, we adapt a frequency-domain method, f-k migration, for solving the inverse NLOS problem.

Video

Abstract

Imaging objects outside a camera’s direct line of sight has important applications in robotic vision, remote sensing, and many other domains. Time-of-flight-based non-line-of-sight (NLOS) imaging systems have recently demonstrated impressive results, but several challenges remain. Image formation and inversion models have been slow or limited by the types of hidden surfaces that can be imaged. Moreover, non-planar sampling surfaces and non-confocal scanning methods have not been supported by efficient NLOS algorithms. With this work, we introduce a wave-based image formation model for the problem of NLOS imaging.

Inspired by inverse methods used in seismology, we adapt a frequency-domain method, f−k migration, for solving the inverse NLOS problem. Unlike existing NLOS algorithms, f−k migration is both fast and memory efficient, it is robust to specular and other complex reflectance properties, and we show how it can be used with non-confocally scanned measurements as well as for non-planar sampling surfaces. f−k migration is more robust to measurement noise than alternative methods, generally produces better quality reconstructions, and is easy to implement. We experimentally validate our algorithms with a new NLOS imaging system that records room-sized scenes outdoors under indirect sunlight, and scans persons wearing retroreflective clothing at interactive rates.

Slides

Reconstructing Hidden Objects

Experimental Setup: The imaging system records the time it takes for laser light to scatter off the wall, reflect off the hidden bunny, and return to the wall. By acquiring these timing measurements for different laser positions on the wall, the 3D geometry of the hidden object can be reconstructed.

Reconstructing Hidden Objects

Room-sized Reconstruction: We capture measurements of this hidden room-sized scene and accurately reconstruct the geometry

Interactive Capture

Interactive Capture: By rapidly scanning the wall, we can capture the movement of a person in a retroreflective suit at interactive rates of 4 fps. The reconstruction clearly shows the pose of the actor, imaged from around the corner.

Outdoor Results

Outdoor Result: We can reconstruct a hidden scene outdoors by scanning the side of a stone building. The stone patterning of the wall is visible in the indirect measurements, but the reconstruction still robustly recovers the geometry.

Captured Results

Comparison to non-line-of-sight imaging using phasor-field virtual wave optics by Liu et al. (2019). The code to run this comparison is available here.

Hardware Prototype

Hardware Prototype: Our hardware prototype captures confocal NLOS measurements with a laser that is 10,000 more powerful than previous systems. The sensor generates timestamps of arriving photons at picosecond time intervals, allowing us to precisely localize features of the hidden geometry.

Code and Datasets

It took a lot of effort to build and calibrate this hardware setup and to capture these data. Feel free to use the datasets in your own projects, but please acknowledge our work by citing the following papers:

• Matthew O’Toole, Felix Heide, David B. Lindell, Kai Zang, Steven Diamond, and Gordon Wetzstein. 2017. Reconstructing transient images from single-photon sensors. In Proc. CVPR.

• Matthew O’Toole, David B. Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 7696, 338.

• Felix Heide, Matthew O’Toole, Kai Zang, David B. Lindell, Steven Diamond, and Gordon Wetzstein. 2018. Non-line-of-sight Imaging with partial occluders and surface normals. ACM Trans. Graph.

• David B. Lindell, Gordon Wetzstein, and Matthew O’Toole. 2019. Wave-based non-line-of-sight Imaging using fast f−k migration. ACM Trans. Graph. (SIGGRAPH) 38, 4, 116.

Acknowledgments

This project was supported by a Stanford Graduate Fellowship, an NSF CAREER Award (IIS 1553333), a Terman Faculty Fellowship, a Sloan Fellowship, by the KAUST Office of Sponsored Research through the Visual Computing Center CCF grant, the Center for Automotive Research at Stanford (CARS), the DARPA REVEAL program, and a PECASE from the ARO.

Citation

@article{Lindell:2019:Wave,
author = {David B. Lindell and Gordon Wetzstein and Matthew O’Toole},
title = {Wave-based non-line-of-sight imaging using fast f--k migration},
journal = {ACM Trans. Graph. (SIGGRAPH)},
volume = {38},
number={4},
pages={116},
year = {2019},
}