Computational mirrors: Blind inverse light transport by deep matrix factorization

• Main Authors: Aittala, Miika (Author), Sharma, Prafull (Author), Murmann, Lukas (Author), Yedidia, Adam B. (Author), Wornell, Gregory W. (Author), Freeman, William T (Author), Durand, Frederic (Author)
• Other Authors: Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory (Contributor), Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science (Contributor)
• Format: Article
• Language: English
• Published: Morgan Kaufmann Publishers, 2021-09-09T15:44:21Z.
• Subjects: Article
• Online Access: Get fulltext

We recover a video of the motion taking place in a hidden scene by observing changes in indirect illumination in a nearby uncalibrated visible region. We solve this problem by factoring the observed video into a matrix product between the unknown hidden scene video and an unknown light transport matrix. This task is extremely ill-posed as any non-negative factorization will satisfy the data. Inspired by recent work on the Deep Image Prior, we parameterize the factor matrices using randomly initialized convolutional neural networks trained in a one-off manner, and show that this results in decompositions that reflect the true motion in the hidden scene.
United States. Defense Advanced Research Projects Agency (Contract HR0011-16-C-0030)
National Science Foundation (U.S.) (Grant CCF-1816209)