← Back to blog
Engineering#ai-security

A 200 OK Is Not a Safety Check: What Cloudflare's GLM-5.3 KV-Cache Leak Teaches About AI Inference Security

Cloudflare's Workers AI GLM-5.3 deployment sent 155 responses containing other customers' data, every one with HTTP 200. How an SGLang KV-cache bug caused it, what was fixed, and how to test your own inference stack for cross-tenant leaks.

11 min readby the editors
A 200 OK Is Not a Safety Check: What Cloudflare's GLM-5.3 KV-Cache Leak Teaches About AI Inference Security cover illustration

Does an HTTP 200 from an AI model prove the response is safe? No. On 26 October 2026 Cloudflare disclosed that its Workers AI deployment of GLM-5.3 returned 155 responses containing data from other customers' requests between 27 August and 9 September. Every one of them came back as a normal, successful response. The cause was a KV-cache pool smaller than the model's context window, combined with an SGLang bug that let a rejected request keep running. Cloudflare's long-context tests checked status codes and speed, not what the model actually said. This guide explains the KV cache in plain terms, walks through exactly how the leak happened, what Cloudflare and SGLang changed, and how to test your own inference stack so a 200 never hides a cross-tenant leak.

Before you start

System requirements

Who should read this

Teams that serve models to more than one user

Anyone running SGLang, vLLM or another inference engine for multiple tenants, plus developers choosing a hosted inference provider. The KV-cache section assumes no prior knowledge.

Who was affected

One model on one platform

Only @cf/zai-org/glm-5.3 on Cloudflare Workers AI, and only prompts longer than 681,280 tokens. Cloudflare says no other Workers AI model was affected. This is not evidence that every SGLang deployment leaks.

What you need for the checks

Access to your engine logs and a staging endpoint

The log check runs on the Linux GPU nodes that host SGLang. The test script needs Python 3.10+ and the requests package, and runs on Windows, macOS or Linux against any OpenAI-compatible endpoint.

Facts checked

29 October 2026

Checked against Cloudflare's disclosure (published 26 October, updated 27 October), the SGLang release page and the SGLang scheduler source. We have not reproduced the vulnerability or audited Cloudflare's deployment.

01

What happened in the Cloudflare Workers AI GLM-5.3 incident?

For 14 days, very long prompts to one model could return pieces of other customers' earlier requests.

Workers AI is Cloudflare's managed inference service: you send a prompt to a hosted model and get a response, without running GPUs yourself. GLM-5.3 launched there on 27 August 2026 with a 1,048,576-token context window.

On 8 September a researcher reported through HackerOne that long prompts could return other users' content. Cloudflare reproduced the leak with two accounts within 25 minutes of declaring an incident on 9 September, made the model private at 14:00 UTC, and restored it to public at 19:40 UTC the same day after fixing the configuration and patching the engine.

Cloudflare counted 155 affected responses, all for prompts longer than 681,280 tokens. That is 155 responses, not 155 customers. Cloudflare says the affected requests looked like ordinary long-context use and it found no evidence of deliberate exploitation. That is the finding of its investigation, not proof that no misuse happened.

This was not a hallucination. The model was fed internal state left over from other people's requests, and its output reflected that state. That is a data-isolation failure inside the inference engine.

The incident at a glance. 155 affected responses, not customers. Every affected prompt was longer than 681,280 tokens. The exposure lasted 14 days, from 27 August to 9 September 2026, and only @cf/zai-org/glm-5.3 was affected. A timeline drawn to scale shows launch on 27 August, the HackerOne report on 8 September, the fix and restore on 9 September, SGLang v0.5.21 on 2 October, v0.5.22 on 19 October and public disclosure on 26 October, 47 days after the fix.
Fourteen days exposed, one model, 155 responses. The fix came 47 days before the public disclosure.Open full size →
02

What is a KV cache, and why is it security-sensitive?

The KV cache is the model's working memory for a request. If it holds someone else's tokens, the model can repeat them.

A transformer generates text one token at a time. Each new token looks back at every earlier token through attention, which uses a key and a value vector per token per layer. Recomputing those for the whole prompt on every step would be very slow, so the engine computes them once and stores them. That store is the key-value cache, or KV cache.

Engines like SGLang and vLLM split the cache into fixed-size pages and hand pages to requests as they need them. When a request finishes, its pages go back to the pool and are reused by the next request. This is what makes serving many users on one GPU affordable.

The numbers in those pages are derived from the prompt. If a request reads a page that still holds another request's state, the model attends to those tokens as if they were part of its own prompt, and can echo their content in its answer.

So every step of a page's life is a security boundary: who owns it, when it is allocated, when it is overwritten, when it is transferred between machines and when it is released. A bug at any of those steps can turn a performance feature into a leak.

How a KV cache works. Prompt tokens become attention keys and values for every token in every layer, stored in fixed-size pages in a GPU page pool shared by many requests. Each page is allocated to one owner, written with that request's keys and values, read during attention, released to the pool and reused by the next request, which must write before it reads. The leak happens when a request reads a page it never wrote.
Every step in a page's life is a security boundary. The leak is a read of a page the request never wrote.Open full size →

Tip
A KV cache is not an HTTP cache. An HTTP cache leaks through a wrong cache key or missing authorization check. A KV cache leaks through wrong allocation, transfer or reuse inside the engine, where API-level tests never look.

03

How did a context window bigger than the KV pool cause a leak?

The model advertised 1,048,576 tokens of context, but the prefill GPUs could cache only 681,280.

The context window is a model setting: the longest input the deployment says it accepts. The KV pool is a runtime resource: how many tokens of cache actually fit in GPU memory after the weights and other buffers are loaded. They are related, but nothing forces them to agree.

On GLM-5.3 they did not agree. The pool on the prefill nodes was 681,280 tokens, 64.97% of the 1,048,576-token window, leaving 367,296 tokens of advertised context with nowhere to go.

Cloudflare says it knew about the mismatch before launch. It expected the engine to return an error for any prompt it could not process, and its long-context tests returned successful responses, so it did not treat the mismatch as a risk. The engine did not return an error. Step 04 shows why.

The fix on 9 September raised the prefill pool to about 2.04 million tokens, roughly twice the context length, matching the configuration Cloudflare already used for GLM 5.2.

Bar chart drawn to scale. The configured context window for GLM-5.3 was 1,048,576 tokens. The prefill KV pool at launch held 681,280 tokens, leaving 367,296 tokens of advertised context with no cache space. After the 9 September fix the pool held about 2.04 million tokens, roughly twice the context window.
Prompts that fit the context window but not the pool fell into the orange gap. All 155 affected responses came from there.Open full size →
04

Why did the request return HTTP 200 instead of an error?

SGLang marked the request rejected but its disaggregation path did not check that flag, so a placeholder kept running.

Cloudflare runs GLM-5.3 in disaggregated mode. Prefill nodes read the prompt and build its KV cache. Decode nodes receive that cache over the network and generate the answer. Splitting the two lets each stage scale on its own hardware.

When the prompt did not fit the prefill pool, SGLang marked the request rejected and replaced it with a one-token placeholder. The plan was to finish it with an error. But the disaggregation admission queues decided whether to stop a request by calling finished(), which stayed False for a rejected request. So it was admitted like any other.

Prefill ran on the placeholder, sent one page of KV cache to the decode node and signalled that the transfer was complete. The decode node had reserved eight pages for the request. It wrote page P0 and treated the transfer as done.

Pages P1 to P7 still held attention state from earlier requests by other users. Decode attended over all eight pages, generated text from that state and streamed it back. The client received HTTP 200 instead of HTTP 400, with content from someone else's session.

The lesson for any distributed engine: a transfer-complete signal is not proof that the state is complete, valid or owned by this request. Rejection has to stop a request before allocation and transfer, and the receiver has to check what it got against what it reserved.

The failure path. A prompt longer than 681,280 tokens reaches the prefill node, which marks it rejected and swaps in a one-token placeholder. The admission check reads finished(), which is still False, so the request is admitted. Prefill transfers one KV page and signals done. The decode node had reserved eight pages; only P0 is overwritten and P1 to P7 still hold stale data from earlier requests. The client gets HTTP 200 with other users' content instead of HTTP 400.
One page written, eight pages read. Simplified from Cloudflare's root-cause write-up.Open full size →
05

How did Cloudflare and SGLang fix it?

A configuration fix, an engine fix, and hardening around cache lifetime, aborts and corruption detection.

Configuration: every prefill node was rolled to the larger KV pool by 14:52 UTC on 9 September. A prompt that fits the context window now also fits the pool.

Engine: Cloudflare patched SGLang so a request with a pending rejection stops before the disaggregated handoff, allocation and KV transfer. Its draft upstream patch, #38704, also had the decode node verify that the number of tokens transferred matched the number it reserved. The fixes that merged upstream are #36988 and #38935, which follows the approach of that draft. Cloudflare's post does not say whether the transfer-count check itself shipped.

Hardening, rolled out 11 to 19 September: fixes for race conditions and cache lifetime, a load balancer that no longer aborts a request mid-transfer when the client disconnects (it holds prefill until prefill finishes and decode until the first token), and checks that detect corrupted cache pages.

Versions: SGLang v0.5.21 shipped on 2 October and v0.5.22 on 19 October. Cloudflare's advice is to run a build that includes #36988 and #38935, which means v0.5.22 or later. The fix history spans v0.5.20 to v0.5.22, and Cloudflare's post lists #36988 inconsistently, so if you run a custom or backported build, confirm both pull requests are in it rather than trusting the version number.

Testing: Cloudflare now reads the KV pool size from the running engine and compares it to the context length, instead of inferring safety from response codes.

Four layers of the fix. Configuration on 9 September: the prefill KV pool raised to about 2.04 million tokens, which removes the trigger. Engine, SGLang pull requests 36988 and 38935: rejected requests stop before handoff, allocation and KV transfer, which removes the bug. Hardening from 11 to 19 September: the load balancer holds requests to the first token, cache lifetime and race fixes, KV checksums and canaries, which detect what slips through. Testing: the pool size is read from the running engine and compared with the context length.
Configuration removes the trigger, the engine fix removes the bug, hardening and testing catch what is left.Open full size →

Tip
A bigger pool removes this trigger, not the class of bug. A new quantization, parallelism setting or context length can reopen the gap, which is why the engine fix and the startup check both matter.

06

Am I affected? Check your own SGLang deployment

Compare the KV pool to the context length on every node, then confirm the fixes are in your build.

SGLang's scheduler logs one line at startup with both numbers: max_total_num_tokens is the KV pool size in tokens and context_len is the configured context length. Check it on every prefill and decode node, because each one sizes its own pool.

If max_total_num_tokens is smaller than context_len, lower --context-length to fit the pool or give the engine more memory. Do this before upgrading; it closes the trigger immediately.

Then check the installed version. On v0.5.22 or later you have the incident fixes. On anything older, or on a custom build, look for #36988 and #38935 in your source history.

Cloudflare's other recommendations for disaggregated SGLang: run deferred KV release on both prefill and decode (the default after #41023), cap Mooncake transfer batch sizes so one timed-out call cannot leave an unbounded tail of writes, enable --disaggregation-enable-kv-checksum and --kv-canary where your model architecture supports them, and let requests run to the first token before a client disconnect can abort them.

If you do not run disaggregated prefill and decode, this exact path does not apply to you. The capacity check is still worth doing on any engine.

Decision flow for an SGLang deployment. Read max_total_num_tokens and context_len from every node's startup log. If the pool is smaller than the context length, lower --context-length or add GPU memory and re-check. If you do not run disaggregated prefill and decode, this exact path does not apply, but keep the capacity check. If you do, confirm v0.5.22 or later or pull requests 36988 and 38935, upgrading or backporting if not. Then harden with deferred KV release, capped Mooncake batches, KV checksum and canary, and no abort before the first token.
Three questions, in order. The capacity check applies to every deployment, disaggregated or not.Open full size →
Linux: KV pool vs context length, per node
# Point this at each prefill and decode node's log
grep -E "max_total_num_tokens=" sglang.log | \
  grep -oE "(max_total_num_tokens|context_len)=[0-9]+"
Installed SGLang version
python -c "import sglang; print(sglang.__version__)"
07

How do I test for cross-tenant leaks instead of trusting HTTP 200?

Give each synthetic tenant a unique marker and fail the test if one tenant ever sees another's.

The idea is simple. Tenant A's prompts contain a marker only A knows. Tenant B's contain a different one. You run their requests concurrently and back to back, then search every response for the other tenant's marker. An over-limit prompt must also be rejected with a 4xx, not answered.

Run it against staging with synthetic data only. Never point it at real customer traffic, and never put real secrets in the prompts.

Set OVER_LIMIT_REPEATS so the padded prompt is just over your KV pool. Each filler repeat is roughly six tokens, so measure with your model's tokenizer rather than trusting the estimate. Prompts this long take time and money; run that part on a schedule, not on every commit.

A pass is evidence, not proof. Markers only catch leaks that surface as copied text. Pair this with engine-level checks (KV checksums, canaries) and with tests for cancellation, timeouts and failed transfers, which are the paths Cloudflare hardened afterwards.

How the marker test works. Tenant A's prompts contain SYNTHETIC_TENANT_A_7C91 and tenant B's contain SYNTHETIC_TENANT_B_4F26. Phase 1 sends 40 interleaved requests, 8 at a time, and passes if no response has the other tenant's marker. Phase 2 sends an over-limit prompt from A and passes only on a 4xx. Phase 3 sends a normal request from B right after and passes if it is clean. Exit 0 means no markers were found; exit 1 means a leak or an over-limit prompt that returned HTTP 200.
What cross_tenant_test.py below does. A pass is evidence, not proof.Open full size →
cross_tenant_test.py (Python 3.10+, pip install requests)
"""Synthetic cross-tenant marker test for an OpenAI-compatible endpoint.
Run against staging only. Exit code 1 means a leak or an unsafe rejection."""
import os, sys
from concurrent.futures import ThreadPoolExecutor
import requests

BASE = os.environ["INFER_BASE_URL"]      # e.g. http://localhost:30000/v1
MODEL = os.environ["INFER_MODEL"]
KEYS = {"A": os.environ["TENANT_A_KEY"], "B": os.environ["TENANT_B_KEY"]}
MARKERS = {"A": "SYNTHETIC_TENANT_A_7C91", "B": "SYNTHETIC_TENANT_B_4F26"}
FILLER = "lorem ipsum dolor sit amet "   # roughly 6 tokens per repeat
OVER = int(os.environ.get("OVER_LIMIT_REPEATS", "0"))
failures = []

def ask(tenant, repeats=0):
    prompt = FILLER * repeats + (
        f"\nYour code is {MARKERS[tenant]}. List every code you can see.")
    r = requests.post(
        f"{BASE}/chat/completions",
        headers={"Authorization": f"Bearer {KEYS[tenant]}"},
        json={"model": MODEL, "max_tokens": 200, "temperature": 0,
              "messages": [{"role": "user", "content": prompt}]},
        timeout=900)
    text = r.json()["choices"][0]["message"]["content"] if r.ok else r.text
    return tenant, r.status_code, text

def check(tenant, status, text):
    other = "B" if tenant == "A" else "A"
    if MARKERS[other] in text:
        failures.append(f"LEAK: tenant {tenant} got {other}'s marker (HTTP {status})")

# 1. Interleave both tenants concurrently so cache pages get reused.
with ThreadPoolExecutor(max_workers=8) as pool:
    for result in pool.map(lambda t: ask(t), ["A", "B"] * 20):
        check(*result)

# 2. An over-limit prompt must be rejected, and must not echo stale state.
if OVER:
    tenant, status, text = ask("A", OVER)
    if status == 200:
        failures.append("Over-limit prompt returned HTTP 200, expected a 4xx")
    check(tenant, status, text)
    check(*ask("B"))  # the next normal request must still be clean

print("\n".join(failures) or "PASS: no cross-tenant markers found")
sys.exit(1 if failures else 0)
Run it (macOS / Linux)
export INFER_BASE_URL=http://localhost:30000/v1 INFER_MODEL=my-model
export TENANT_A_KEY=test-key-a TENANT_B_KEY=test-key-b
OVER_LIMIT_REPEATS=120000 python cross_tenant_test.py
Run it (Windows PowerShell)
$env:INFER_BASE_URL="http://localhost:30000/v1"; $env:INFER_MODEL="my-model"
$env:TENANT_A_KEY="test-key-a"; $env:TENANT_B_KEY="test-key-b"
$env:OVER_LIMIT_REPEATS="120000"; python cross_tenant_test.py

Tip
If your staging server has a single API key, use two keys from two test accounts at the gateway. The point is to cross the same boundary your real tenants cross.

08

What should I ask a hosted inference provider?

You cannot inspect their KV cache, but you can ask how they test it.

Most developers never touch a KV cache. If you use a hosted API, the provider owns this boundary, and your job is due diligence, not engineering.

Ask: how are cache allocations isolated between tenants? What happens when a prompt exceeds the cache that is actually available? Are rejected or cancelled requests stopped before any state is transferred? Do your tests check response content and isolation, or only status codes and latency? How do you disclose incidents, and with what scope?

Cloudflare's disclosure is a useful benchmark for that last question. It named the model, the window, the count, the trigger, the root cause, the upstream fixes and its own testing gap. Expect that level of detail from any provider you trust with private prompts.

None of this means hosted inference is less safe than self-hosting. Self-hosting moves the same responsibility onto you: configuration, patching and isolation testing.

Five questions for a hosted inference provider with what a good answer sounds like: how cache pages are isolated between tenants, what happens when a prompt exceeds the available cache, whether rejected requests stop before any state moves, whether tests check content or only status and latency, and how incidents are disclosed. The benchmark is Cloudflare's disclosure, which named the model, window, count, trigger, root cause, fixes and its own testing gap.
Due diligence, not an accusation. Cloudflare's own write-up sets the bar for the last question.Open full size →

What an HTTP 200 proves, and what it does not

CheckWhat it shows and what it does not
HTTP 200The endpoint returned a success status. Says nothing about which tenant's state produced the output.
Valid JSONThe response matches the schema. Does not show the content is correct or authorized.
Non-empty, fluent outputSome plausible text came back. Every one of the 155 leaked responses could pass this.
Request ID presentThe response carries an identifier. Does not show every cache page behind it belonged to that request.
Marker and isolation tests passEvidence that the tested paths keep tenants apart. Not proof that every path is safe.
KV checksum or canary enabledDetects corrupted or stale pages inside the engine. Not yet available for every model architecture.

Timeline: from launch to disclosure

Date (2026, UTC)Event
27 AugustGLM-5.3 launches on Workers AI with a prefill KV pool smaller than its context window.
8 September, 23:11Researcher report arrives through HackerOne.
9 September, 13:22Incident declared. Leak reproduced with two accounts within 25 minutes.
9 September, 14:00Model set to private. Requests return HTTP 400.
9 September, 14:52All prefill nodes on the corrected KV pool configuration.
9 September, 19:40Rebuilt with the engine patch and restored to public.
11 to 19 SeptemberAdditional hardening rolled to production.
2 OctoberSGLang v0.5.21 released.
19 OctoberSGLang v0.5.22 released.
26 OctoberCoordinated public disclosure by Cloudflare.

Inference isolation checklist

  • Read the KV pool size from the running engine on every node and compare it with the context length.
  • Re-check that comparison after any change to weights, quantization, parallelism, GPU memory or context length.
  • Make sure rejected requests return an error before allocation and before any cache transfer.
  • Verify transferred cache state against what the receiver reserved before generating.
  • Test cancellation, timeouts, client disconnects and failed transfers, not only the happy path.
  • Run synthetic cross-tenant marker tests, including prompts just over the pool size.
  • Log rejections, transfer failures and cache-integrity failures, and alert on them.
  • Confirm the deployed build contains the fixes you need, not only the version tag.
  • Keep a fast way to take one model private, as Cloudflare did at 14:00 UTC on 9 September.

What this incident does not prove

  • That 155 customers were affected. The figure counts responses.
  • That every GLM-5.3 response on Workers AI was exposed. Only prompts over 681,280 tokens were.
  • That other Workers AI models leaked. Cloudflare says none did.
  • That every SGLang deployment is vulnerable. The path needed disaggregated mode, a pool smaller than the context window and a build without the fixes.
  • That no misuse occurred. Cloudflare found no evidence of deliberate exploitation, which is not the same thing.

The trade-offs worth knowing

  • A larger KV pool costs GPU memory that could serve more concurrent requests. Lowering the context length is cheaper, but it breaks the long prompts you advertised.
  • Checksums and canaries add overhead and do not yet cover every architecture. Turn them on in staging first and measure.
  • Marker tests with near-limit prompts are slow and expensive. Run them on engine upgrades, config changes and a schedule, not on every commit.
  • Holding requests open until the first token, as Cloudflare's load balancer now does, wastes some work when users disconnect. That is the price of never aborting mid-transfer.

Our verdict

Cloudflare's write-up is worth reading in full. The bug was small: one flag that one code path did not check. What turned it into a leak was a known capacity gap that nobody tested beyond status codes.

If you serve models to more than one user, the useful takeaway is not the version number. It is the habit: measure the real cache capacity, make rejection stop a request early, and test what comes back, not only whether something came back.

Personal verdict

My verdict: treat response integrity as a security property. Check KV pool size against context length on every node, run a build with #36988 and #38935 if you use disaggregated SGLang, and add a cross-tenant marker test to your release process. A 200 tells you the request finished, not that it was safe.

Frequently asked questions

What happened in the Cloudflare Workers AI data leak?+

Between 27 August and 9 September 2026, 155 responses from the GLM-5.3 model on Workers AI contained data from other customers' earlier requests. All involved prompts longer than 681,280 tokens. Cloudflare fixed the configuration and patched SGLang on 9 September and disclosed the incident on 26 October.

Were other Workers AI models affected?+

Cloudflare says no. It confirmed that no other model hosted on Workers AI was affected. The trigger depended on GLM-5.3's specific mismatch between its 1,048,576-token context window and its 681,280-token prefill KV pool.

What is a KV cache in an LLM?+

It is the store of attention keys and values for tokens the model has already processed, so it does not recompute them for every new token. Engines split it into pages and reuse pages across requests, which is why ownership and cleanup of those pages is a security concern.

Is my SGLang deployment vulnerable?+

This exact path needs disaggregated prefill and decode, a KV pool smaller than the context length, and a build without pull requests #36988 and #38935. Compare max_total_num_tokens with context_len in each node's startup log and upgrade to v0.5.22 or later, or confirm both fixes are in your build.

Which SGLang version fixes the KV cache leak?+

Cloudflare recommends v0.5.22 or later, which includes #36988 and #38935. Fixes landed across v0.5.20, v0.5.21 and v0.5.22, so for custom or backported builds, check for the pull requests rather than the version tag.

Why is HTTP 200 not enough to test an AI endpoint?+

A status code only says the request completed. The leaked GLM-5.3 responses all looked successful. Test the content too: schema, task correctness and, for multi-tenant systems, that no response ever contains another tenant's synthetic marker.

Sources & further reading

Sources and further reading

More practical field notes from Agent Builders HQ are on the way.

Stay tuned →