Forgotten AI Research Solved The Problem Photoshop Never Could!

Quick Overview

A new AI relighting technique allows for the manipulation of light sources in 2D photographs, a capability previously exclusive to 3D modeling, by de-lighting the original image, converting it into a 3D model, and then using a neural renderer to reconstruct a photorealistic image with new lighting, all processed in under three seconds.

Key Points: A new AI technique allows for the relighting of 2D photographs, a capability previously impossible without 3D modeling. The process involves de-lighting the image, converting it to a 3D model, and using a neural renderer to reconstruct it with new lighting. The neural renderer is trained by iteratively adjusting lights in a 3D scene and comparing the output to the original photo until the scene and lighting accurately explain the image. Users can change the time of day, add or remove lights, and create shadows in 2D photos. The entire process runs in under three seconds: "all this runs in within three seconds." Limitations include blocky artifacts, less-than-fantastic 3D geometry resolution, and issues with complex materials like skin.

Context: The video discusses a new AI research paper that addresses a long-standing limitation in digital photography: the inability to easily relight 2D photographs after they have been taken. Unlike 3D modeling environments where lighting is easily adjustable, traditional 2D photo editing software, even with AI advancements, could only perform minor edits to lighting. This new technique aims to bridge that gap by enabling dynamic manipulation of light sources within static images.

Detailed Analysis

This video introduces a novel AI relighting technique that enables users to change lighting in 2D photographs, a feat previously impossible without 3D modeling software. The process involves three main steps: first, de-lighting the original 2D photograph using existing methods. Second, converting the de-lited 2D image into a 3D model, which, while initially rough and lacking detail, serves as the foundation. The crucial third step is employing a neural renderer, trained on thousands of image-rendering pairs, to transform the rough 3D rendering into a photorealistic image that accurately reflects the new lighting conditions. This neural renderer is trained by iteratively adjusting lights in a 3D scene and comparing the rendered output to the target photograph, refining the process until the 3D scene and lighting explain the original photo. The result is the ability to alter time of day, add or remove light sources, and even cast shadows, all within seconds. While the technique is fast, processing in under three seconds, it has limitations, including blocky artifacts with moving lights, less-than-fantastic 3D geometry resolution, issues with lights in unexpected places, and difficulties with specular highlights and complex materials like skin. Despite these limitations, the research represents a significant advancement, transforming static photographs into dynamic, editable worlds and empowering artists to manipulate reality after the shot.