Deepseek just killed LLMs

Quick Overview

The video analyzes recent developments in AI, focusing on DeepSeek-OCR's optical context compression for LLMs, Andrej Karpathy's critique of tokenizers and promotion of image-based inputs, Google AI's quantum computing breakthrough, and the controversy surrounding the 'Definition of AGI' paper's fabricated citations, all while highlighting the rapid progress and associated safety/infrastructure concerns in the AI field.

Key Points: DeepSeek-OCR achieves up to 20x compression of visual contexts while maintaining 97% OCR accuracy at context lengths under 10k tokens, demonstrating efficient optical context compression for LLMs. Andrej Karpathy champions image-based inputs over text tokens, arguing text tokenizers are wasteful and introduce historical baggage like security/jailbreak risks, suggesting pure image inputs are superior. Google AI announced a major quantum computing breakthrough, demonstrating a quantum computer running a verifiable algorithm 13,000x faster than leading classical supercomputers. The viral 'Definition of AGI' paper faced scrutiny after Michael Saxon pointed out it contained fake citations, which co-author Dan Hendrycks attributed to an incorrect conversion from a Google Doc to BibTeX. The video contrasts these advancements with infrastructure scaling, noting that training models like the C2S-Scale 27B costs as little as $100, but scaling laws suggest costs will increase quadratically with sequence length. The video also references the C2S-Scale 27B model's success in discovering a novel cancer therapy pathway, validating the power of scaling laws even in biological discovery. The discussion touches on the debate between token-based and pixel-based inputs, with Karpathy strongly favoring pixels due to tokenization's inherent issues like producing 'weird tokens' for emojis.

Context: This video provides a commentary and digest of recent, high-profile news and research papers in the field of Artificial Intelligence, primarily focusing on Large Language Models (LLMs) and multimodal AI. Key subjects include the release of DeepSeek-OCR, Andrej Karpathy's ongoing thoughts on model input modalities (text vs. pixels), major announcements from Google AI regarding quantum computing, and a controversy surrounding a widely circulated AGI definition paper that contained fake citations. The speaker analyzes these topics by referencing the original tweets and research abstracts.