Is Nano Banana Pro a Low-Level Vision All-Rounder? A Comprehensive Eval on 14 Tasks and 40 Datasets

Quick Overview

Nano-Banana Pro (NB Pro) demonstrates superior performance in visual quality metrics, achieving incredibly high perceptual scores, but it struggles significantly with semantic fidelity and identity preservation, leading to severe distortions like identity swaps and creation of artifacts when dealing with complex inputs like text or scenes requiring high factual accuracy.

Key Points: NB Pro achieved near-perfect perceptual quality scores (PSNR/SSIM) on synthetic tasks, outperforming dedicated models like SRGAN. The model struggles severely with identity preservation, frequently swapping identities between subjects in complex visual scenes (02:59). When restoring text, NB Pro introduced significant errors, showing an inability to maintain factual accuracy, especially with complex characters (06:18). The model failed catastrophically on reflection removal tasks, creating artifacts like bright light reflections on glass where none should exist (07:47). NB Pro's performance on tasks requiring high factual accuracy, such as fixing motion blur or noise, resulted in artifacts or complete loss of semantic content (06:21). The core issue identified is the model's tendency to prioritize visual aesthetics (high perceptual quality) over adherence to the ground truth of the original scene or identity (07:55, 09:28).

Context: This video presents a comprehensive evaluation of Google's latest generative text-to-image model, Nano-Banana Pro (NB Pro), which is built upon the Gemini 1.5 Pro engine. The evaluation tests NB Pro across 14 different technical tasks and 40 datasets, focusing on comparing its output quality against both traditional image restoration techniques and other generative models, particularly concerning perceptual fidelity versus semantic accuracy.

Detailed Analysis

The evaluation of Nano-Banana Pro (NB Pro) reveals a dual nature: it excels in metrics related to visual appeal but fails significantly in maintaining semantic fidelity and factual accuracy. On tasks like image enhancement and restoration, NB Pro achieved remarkably high perceptual scores (PSNR/SSIM), often surpassing dedicated baseline models like SRGAN, suggesting its output looks aesthetically pleasing (04:51, 07:51). However, this aesthetic quality comes at a high cost. When tested on tasks involving complex scenes, like restoring an underwater scene or fixing motion blur, the model generated artifacts, ghostly elements, or completely failed to recover the correct semantic content (06:21). A critical failure point was identity preservation; in complex image fusion tasks, NB Pro swapped identities between subjects (02:59). Furthermore, when presented with text, the model struggled to maintain character fidelity, leading to nonsensical output. The core takeaway is that NB Pro prioritizes creating a visually coherent scene over accurately representing the input's factual or semantic details, making it unreliable for forensic or high-stakes applications where accuracy is paramount (09:12, 10:43). The authors suggest that better prompt engineering or few-shot learning might mitigate some issues, but the fundamental trade-off remains.