did rust take down the internet?

Quick Overview

The Rust function did not take down the internet, but a recent Cloudflare outage was caused by a change in their underlying ClickHouse query behavior that led to an over-allocation of memory within the Bot Management feature, causing a system panic because the feature file exceeded a hardcoded limit of 200 features set for performance optimization.

Key Points: The Cloudflare outage stemmed from a change in underlying ClickHouse query behavior which caused Bot Management feature files to contain duplicate rows, effectively doubling the feature count. The Bot Management system has a hardcoded limit of 200 machine learning features, and the duplicated data caused this limit to be exceeded, leading to system panic. The resulting panic occurred when the Rust code called on an error value within the , causing 5xx errors across services relying on the core proxy, including Workers KV and Cloudflare Access. The speaker critiques the Rust function name because it panics, contrasting it with non-panicking alternatives like or , suggesting would be clearer. The memory preallocation limit was set to 200 features, far above the usual 60 features, as a performance optimization measure. Other systems relying on the core proxy, like Workers KV and Cloudflare Access, were impacted, leading to reduced error rates and login issues for Dashboard/Turnstile users.

Context: This video analyzes the cause of a recent, significant internet outage affecting Cloudflare services, contrasting the popular speculation that Rust's error handling mechanisms were at fault (specifically the function) with the actual technical explanation provided by Cloudflare in their incident report. The speaker uses Rust code examples to explain why the naming convention of panicking functions like can be confusing compared to non-panicking alternatives.

Detailed Analysis

The video explains that while some initially blamed Rust's function for the Cloudflare outage, the actual cause was a performance-related memory allocation issue within Cloudflare's Bot Management system. A change in the underlying ClickHouse query behavior resulted in feature files containing duplicate rows, effectively doubling the number of features processed, which pushed the count over a hardcoded limit of 200 features set as a performance guardrail. This limit was set high (200 vs. the usual 60) due to performance reasons, but the bad file caused the system to panic when the Rust code attempted to call on an error value within the . This resulted in 5xx errors for traffic relying on the core proxy, impacting services like Workers KV and Cloudflare Access. The speaker then pivots to critique the Rust convention, arguing that is a poor name for a panicking function, especially since safer alternatives like exist, suggesting would improve clarity for developers.