The hidden information in your photos | Nikhil Behari | TEDxMIT

Quick Overview

Nikhil Behari demonstrates how mobile Time-of-Flight (ToF) LiDAR sensors, like those in modern smartphones, capture depth information by timing the return of emitted laser pulses, enabling AI systems to infer three-dimensional scene geometry, including hidden details like reflections and shadows, which are otherwise invisible to standard 2D photography.

Key Points: Over 5 billion pictures are taken daily using phones, with over 90% captured by devices in our pockets. Behari's research at MIT and NASA focuses on designing new AI systems that can use subtle visual cues like shadows and reflections to infer hidden 3D information from standard photos. The Time-of-Flight (ToF) LiDAR sensor on phones emits hundreds of laser pulses and measures the time delay of photon returns to calculate distance to objects, generating a depth map. Both shadows and reflections, often ignored or treated as noise, contain valuable hidden information about the 3D world that AI systems can be trained to utilize. The research shows that by analyzing the photon returns, AI can reconstruct 3D models of environments from satellite imagery and even infer details from reflections captured in photos of glossy objects. The demonstration included comparing a blurred LiDAR result with a sharper 3D model derived from the same single photon data, showcasing the power of their 'Blurred LiDAR for Sharper 3D' technique. The speaker invited the audience to examine their own phone camera rolls to see how much information they capture daily that current AI systems fail to utilize.

Context: Nikhil Behari, a graduate student at MIT's Camera Culture Group and a NASA graduate research fellow, presents his work on developing AI systems capable of extracting richer 3D geometric information from standard 2D images by leveraging subtle visual cues like shadows and reflections. The talk, presented at TEDxMIT, uses personal photos from his time in Boston to illustrate how the LiDAR sensor on modern phones functions and how his research aims to move beyond direct depth sensing to interpret these complex visual phenomena.