How one computer file accidentally took down 20% of the internet yesterday

Yesterday’s outage showed how dependent the modern web is on a handful of core infrastructure providers.

In fact, it’s so dependent that a single configuration error made large parts of the internet totally unreachable for several hours.

Many of us work in crypto because we understand the dangers of centralization in finance, but the events of yesterday were a clear reminder that centralization at the internet’s core is just as urgent a problem to solve.

The obvious giants like Amazon, Google, and Microsoft run enormous chunks of cloud infrastructure.

But equally critical are firms like Cloudflare, Fastly, Akamai, DigitalOcean, and CDN (servers that deliver websites faster around the world) or DNS (the “address book” of the internet) providers such as UltraDNS and Dyn.

Most people barely know their names, yet their outages can be just as crippling, as we saw yesterday.

To start with, here’s a list of companies you may never have heard of that are critical to keeping the internet running as expected.

What happened yesterday

Yesterday’s culprit was Cloudflare, a company that routes almost 20% of all web traffic.

It now says the outage started with a small database configuration change that accidentally caused a bot-detection file to include duplicate items.

That file suddenly grew beyond a strict size limit. When Cloudflare’s servers tried to load it, they failed, and many websites that use Cloudflare began returning HTTP 5xx errors (error codes users see when a server breaks).

Here’s the simple chain:

A Small Database Tweak Sets Off a Big Chain Reaction.

The trouble began at 11:05 UTC when a permissions update made the system pull extra, duplicate information while building the file used to score bots.

That file normally includes about sixty items. The duplicates pushed it past a hard cap of 200. When machines across the network loaded the oversized file, the bot component failed to start, and the servers returned errors.

According to Cloudflare, both the current and older server paths were affected. One returned 5xx errors. The other assigned a bot score of zero, which could have falsely flagged traffic for customers who block based on bot score (Cloudflare’s bot vs. human detection).

Diagnosis was tricky because the bad file was rebuilt every five minutes from a database cluster being updated piece by piece.

If the system pulled from an updated piece, the file was bad. If not, it was good. The network would recover, then fail again, as versions switched.

According to Cloudflare, this on-off pattern initially looked like a possible DDoS, especially since a third-party status page also failed around the same time. Focus shifted once teams linked errors to the bot-detection configuration.

By 13:05 UTC, Cloudflare applied a bypass for Workers KV (login checks) and Cloudflare Access (authentication system), routing around the failing behavior to cut impact.

The main fix came when teams stopped generating and distributing new bot files, pushed a known good file, and restarted core servers.

Cloudflare says core traffic began flowing by 14:30, and all downstream services recovered by 17:06.

The failure highlights some design tradeoffs.

Cloudflare’s systems enforce strict limits to keep performance predictable. That helps avoid runaway resource use, but it also means a malformed internal file can trigger a hard stop instead of a graceful fallback.

Because bot detection sits on the main path for many services, one module’s failure cascaded into the CDN, security features, Turnstile (CAPTCHA alternative), Workers KV, Access, and dashboard logins. Cloudflare also noted extra latency as debugging tools consumed CPU while adding context to errors.

On the database side, a narrow permissions tweak had wide effects.

The change made the system “see” more tables than before. The job that builds the bot-detection file did not filter tightly enough, so it grabbed duplicate column names and expanded the file beyond the 200-item cap.

The loading error then triggered server failures and 5xx responses on affected paths.

Impact varied by product. Core CDN and security services threw server errors.

Workers KV saw elevated 5xx rates because requests to its gateway passed through the failing path. Cloudflare Access had authentication failures until the 13:05 bypass, and dashboard logins broke when Turnstile could not load.

Cloudflare Email Security temporarily lost an IP reputation source, reducing spam detection accuracy for a period, though the company said there was no critical customer impact. After the good file was restored, a backlog of login attempts briefly strained internal APIs before normalizing.

The timeline is straightforward.

The database change landed at 11:05 UTC. First customer-facing errors appeared around 11:20–11:28.

Teams opened an incident at 11:35, applied the Workers KV and Access bypass at 13:05, stopped creating and spreading new files around 14:24, pushed a known good file and saw global recovery by 14:30, and marked full restoration at 17:06.

According to Cloudflare, automated tests flagged anomalies at 11:31, and manual investigation began at 11:32, which explains the pivot from suspected attack to configuration rollback within two hours.

The numbers explain both cause and containment.

A five-minute rebuild cycle repeatedly reintroduced bad files as different database pieces updated.

A 200-item cap protects memory use, and a typical count near sixty left comfortable headroom, until the duplicate entries arrived.

The cap worked as designed, but the lack of a tolerant “safe load” for internal files turned a bad config into a crash instead of a soft failure with a fallback model. According to Cloudflare, that’s a key area to harden.

Cloudflare says it will harden how internal configuration is validated, add more global kill switches for feature pipelines, stop error reporting from consuming large CPU during incidents, review error handling across modules, and improve how configuration is distributed.

The company called this its worst incident since 2019 and apologized for the impact. According to Cloudflare, there was no attack; recovery came from halting the bad file, restoring a known good file, and restarting server processes.