# Researchers Are Getting Really Creative Training LLMs

Source: https://www.youtube.com/watch?v=sgIB7l6hW3Q
Recap page: https://rapidrecap.app/video/sgIB7l6hW3Q
Generated: 2025-10-16T14:34:45.426+00:00

---
## Quick Overview

The video primarily contrasts traditional Next Token Prediction (NTP) with Multi-Token Prediction (MTP) and Token Order Prediction (TOP) in training Large Language Models (LLMs), detailing MTP's initial success in code benchmarks but its eventual pitfalls, and concluding that TOP, which is a softer auxiliary objective, performs better and is more aligned with non-greedy inference, while also promoting the cloud platform Sevalla by Kinsta for its developer-centric features and simple deployment.

**Key Points:**
- Multi-Token Prediction (MTP) showed initial gains on code benchmarks (e.g., 17% more on MBPP for 13B models) but performed worse than NTP on smaller models and failed to generalize well on NLP tasks.
- Token Order Prediction (TOP) is introduced as an alternative to MTP that uses an additional head to predict the order of future tokens, resulting in improvements across various NLP benchmarks like HumanEval and MBPP, especially for larger models.
- The authors find that TOP's auxiliary objective is 'softer' than MTP's, as it only needs to assign the highest score to the next token, unlike MTP which penalizes incorrect ordering too harshly.
- At inference time, TOP models revert to using only the NTP head, making the final architecture equivalent to the original transformer, unlike MTP which requires extra computation/heads.
- The video concludes by promoting Sevalla by Kinsta, a cloud platform that combines developer-centric UX (like Render/Railway) with global reach (Fly.io), edge speed (Vercel), and managed services (Platform.sh) for easy deployment of apps, storage, and databases.
- Sevalla offers unlimited resources, unlimited free traffic, and unlimited parallel builds, contrasting with the complexity and potential surprise billing of other platforms.

![Screenshot at 00:12: Comparison demonstrating that Multi-Token Prediction \(MTP\) achieves a full, correct sentence completion \("The cat sat on the mat and stared blankly at the wall."\) while Next Token Prediction \(NTP\) is shown failing to complete the sentence in one step, illustrating the core concept difference.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-00-12.png)

**Context:** This video explains and compares advanced LLM training techniques: Next Token Prediction (NTP), Multi-Token Prediction (MTP), and Token Order Prediction (TOP), based on recent research papers. The discussion highlights the trade-offs between predicting multiple tokens simultaneously (MTP/TOP) versus the standard one-token-at-a-time approach (NTP), focusing on performance gains, training difficulty, and inference costs. The latter half transitions to promoting Sevalla by Kinsta, a cloud platform that aims to simplify deployment by integrating features commonly found across leading cloud providers.

## Detailed Analysis

The video begins by contrasting the standard LLM training method, Next Token Prediction (NTP), where models predict one token sequentially, with Multi-Token Prediction (MTP), where models predict $N$ future tokens simultaneously using $N$ output heads. While MTP showed significant gains on code benchmarks for larger models (e.g., 13B parameters saw 17% more on MBPP), it performed worse than NTP on smaller models and failed to generalize on general NLP tasks, proving difficult to train. The video then introduces Token Order Prediction (TOP) as an improvement over MTP, which also predicts multiple future tokens but uses an additional head (TDP Head) to learn the relative order of those tokens, leading to better performance across the board, especially for larger models, without the high training cost of MTP. Critically, at inference, TOP models discard the order-prediction head and rely only on the NTP head, simplifying deployment compared to MTP. The research suggests TOP is a 'softer' auxiliary objective than MTP because it focuses on relative order rather than forcing exact future token prediction, leading to better learning. The video concludes by pivoting to a promotion for Sevalla by Kinsta, a cloud platform designed to offer an easy, unified experience combining the developer UX of Render/Railway, the global reach of Fly.io, Vercel's edge speed, and managed services like Platform.sh, all while promising unlimited resources and traffic without complex plans or surprise bills.

### LLM Prediction Methods

- Next Token Prediction (NTP) predicts one word sequentially
- Multi-Token Prediction (MTP) predicts N tokens at once, showing mixed results
- Token Order Prediction (TOP) predicts N tokens and their relative order, showing consistent gains

### MTP Performance Analysis

- 13B models saw 17% more on MBPP but smaller models (under 1B) showed performance decreases compared to NTP baseline

### TOP Methodology

- TOP uses an additional Token Dependency Prediction (TDP) Head alongside the NTP Head, with the final loss being a sum of both objectives (L_total = L_NTP + L_TOP)

### TOP vs MTP Inference

- TOP models remove the TDP Head at inference, becoming equivalent to the original NTP architecture, while MTP requires more computation.

### Sevalla Platform Features

- Offers Application hosting, Object storage ($0.02/GB), Database hosting, and Static site hosting (Free tier available)

### Sevalla Benefits

- Unlimited resources (usage-based pricing), unlimited free traffic, unlimited parallel builds, and enterprise-grade security certifications (SOC II Type 2, ISO 27001, GDPR, CCPA)

![Screenshot at 00:05: Illustration contrasting NTP predicting one token at a time versus MTP predicting multiple tokens \(e.g., 4 tokens\) simultaneously.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-00-05.png)
![Screenshot at 00:21: Graphic highlighting the Multi-Token Prediction \(MTP\) concept, which involves multiple output heads operating on top of a shared model trunk.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-00-21.png)
![Screenshot at 00:28: Table 1 showing that for 7B models, MTP largely outperforms NTP across MBPP, HumanEval, and APPS/Intro benchmarks, particularly on code tasks.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-00-28.png)
![Screenshot at 00:33: Slide contrasting NTP \(Next Token Prediction\) which learns one word at a time, versus MTP which predicts 4 tokens at once, showing NTP's sequential nature.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-00-33.png)
![Screenshot at 01:23: Chart showing that for 7B models on MBPP, MTP provided a +4.5 gain over the baseline, while smaller models \(0.3B, 0.6B\) saw performance drops \(-1.7 baseline loss\).](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-01-23.png)
![Screenshot at 01:40: Blackboard stating "There is no universal best amount of heads for MTP," emphasizing that the optimal window size depends on input data distribution.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-01-40.png)
![Screenshot at 04:45: Spider-Man pointing meme comparing MTP \(Multi-Token Prediction\) and TOP \(Text Order Prediction\), implying they are similar concepts.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-04-45.png)
![Screenshot at 06:55: Diagram illustrating the TOP architecture, showing the output of the Transformer Layers feeding into both the NTP Head and the TDP Head, which calculates order rankings.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-06-55.png)
![Screenshot at 07:43: Blackboard stating "TOP = Softer auxiliary objective than MTP," indicating TOP's less restrictive training goal.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-07-43.png)
![Screenshot at 07:55: Final slide showing the creator thanking supporters and promoting the Sevalla platform.](https://ss.rapidrecap.app/screens/sgIB7l6hW3Q/00-07-55.png)