# did rust take down the internet?

Source: https://www.youtube.com/watch?v=nVu4qrNzyqk
Recap page: https://rapidrecap.app/video/nVu4qrNzyqk
Generated: 2025-11-20T17:44:14.108+00:00

---
## Quick Overview

The Rust `unwrap()` function did not take down the internet, but a recent Cloudflare outage was caused by a change in their underlying ClickHouse query behavior that led to an over-allocation of memory within the Bot Management feature, causing a system panic because the feature file exceeded a hardcoded limit of 200 features set for performance optimization.

**Key Points:**
- The Cloudflare outage stemmed from a change in underlying ClickHouse query behavior which caused Bot Management feature files to contain duplicate rows, effectively doubling the feature count.
- The Bot Management system has a hardcoded limit of 200 machine learning features, and the duplicated data caused this limit to be exceeded, leading to system panic.
- The resulting panic occurred when the Rust code called `Result::unwrap()` on an error value within the `fl2_worker_thread`, causing 5xx errors across services relying on the core proxy, including Workers KV and Cloudflare Access.
- The speaker critiques the Rust function name `.unwrap()` because it panics, contrasting it with non-panicking alternatives like `.unwrap_or()` or `.unwrap_or_default()`, suggesting `.or_panic()` would be clearer.
- The memory preallocation limit was set to 200 features, far above the usual ~60 features, as a performance optimization measure.
- Other systems relying on the core proxy, like Workers KV and Cloudflare Access, were impacted, leading to reduced error rates and login issues for Dashboard/Turnstile users.

![Screenshot at 00:03: The DownDetector page showing a massive spike in user reports indicating problems at Cloudflare, visualizing the scale of the recent widespread service outage.](https://ss.rapidrecap.app/screens/nVu4qrNzyqk/00-00-03.png)

**Context:** This video analyzes the cause of a recent, significant internet outage affecting Cloudflare services, contrasting the popular speculation that Rust's error handling mechanisms were at fault (specifically the `.unwrap()` function) with the actual technical explanation provided by Cloudflare in their incident report. The speaker uses Rust code examples to explain why the naming convention of panicking functions like `unwrap` can be confusing compared to non-panicking alternatives.

## Detailed Analysis

The video explains that while some initially blamed Rust's `.unwrap()` function for the Cloudflare outage, the actual cause was a performance-related memory allocation issue within Cloudflare's Bot Management system. A change in the underlying ClickHouse query behavior resulted in feature files containing duplicate rows, effectively doubling the number of features processed, which pushed the count over a hardcoded limit of 200 features set as a performance guardrail. This limit was set high (200 vs. the usual ~60) due to performance reasons, but the bad file caused the system to panic when the Rust code attempted to call `Result::unwrap()` on an error value within the `fl2_worker_thread`. This resulted in 5xx errors for traffic relying on the core proxy, impacting services like Workers KV and Cloudflare Access. The speaker then pivots to critique the Rust convention, arguing that `.unwrap()` is a poor name for a panicking function, especially since safer alternatives like `.unwrap_or()` exist, suggesting `.or_panic()` would improve clarity for developers.

### Cloudflare Outage Cause

- Change in ClickHouse query behavior
- Led to duplicate rows in Bot Management feature files
- Exceeded the 200-feature memory preallocation limit
- Triggered system panic in FL2 Rust code

### Impact of the Panic

- Thread `fl2_worker_thread` panicked on `Result::unwrap()` on an Err value
- Caused 5xx errors for traffic dependent on the bots module
- Affected Workers KV and Cloudflare Access services

### Rust Error Handling Critique

- `.unwrap()` is a terrible name for a panicking function
- Better alternatives include `.unwrap_or()` or `.unwrap_or_default()` which never panic
- Suggests renaming panicking functions to `.or_panic()` for clarity

### Memory Preallocation Limit

- Bot Management system had a limit of 200 features set
- Limit exists to avoid unbounded memory consumption
- The bad file propagated over 200 features, hitting the limit and causing panic

![Screenshot at 00:03: The DownDetector page showing the massive spike in user reports indicating problems at Cloudflare.](https://ss.rapidrecap.app/screens/nVu4qrNzyqk/00-00-03.png)
![Screenshot at 00:09: A diagram showing the architecture where 'You' communicate with 'my-chat-gpt-wrapper-app-saas' through Cloudflare as a reverse proxy.](https://ss.rapidrecap.app/screens/nVu4qrNzyqk/00-00-09.png)
![Screenshot at 01:31: Cloudflare's blog post explaining the issue, showing the underlying ClickHouse SQL query used to retrieve metadata.](https://ss.rapidrecap.app/screens/nVu4qrNzyqk/00-01-31.png)
![Screenshot at 02:21: A snippet of the Rust code showing the pub fn fetch\_features function where the .unwrap\(\) call is located on line 82.](https://ss.rapidrecap.app/screens/nVu4qrNzyqk/00-02-21.png)
![Screenshot at 05:08: The Rust Playground output showing the program panicking due to thread 'main' \(33\) panicked at src/main.rs:17:26: called Result::unwrap\(\) on an 'Err' value: "oops".](https://ss.rapidrecap.app/screens/nVu4qrNzyqk/00-05-08.png)