GLM-5.2 SGLang Repeated Tokens: Diagnose KV Slot 0 Corruption

Evan Brooks

AI Infrastructure Editor

Editorial responsibility: GLM52.ai Editorial Team · Published and editorially reviewed August 30, 2026

How we research and review · Report a correction

Disclosed editorial pen name · AI-generated portrait

A merged SGLang fix does not close every GLM ROCm path

SGLang PR #36003 merged a reserved-slot guard for SGLang-owned MLA writers on August 26, 2026, after the v0.5.18 tag. Its own coverage table excludes the fused AITER GLM writer. The linked AITER pad_slot_id change remains open, and its default preserves legacy slot-zero writes until a downstream caller opts in. Identify the exact writer before deciding that an upgrade fixes the fault.

Original editorial diagram. It explains the write invariant; it is not a SGLang screenshot, GPU trace, model response, benchmark result, package receipt or proof that a deployed kernel contains the fix.

Short answer: repeated tokens can be a visible symptom of the open SGLang MLA KV slot 0 corruption report, but they do not diagnose it by themselves. The stronger test is whether a padding row mapped to physical slot 0 can replace a finite reserved value with NaNs while a valid row still writes to a positive slot. Then the answer depends on the writer: the standard SGLang MLA guard is merged after v0.5.18, while the fused AITER GLM ROCm path remains unresolved in the pinned upstream state checked on August 30, 2026.

Our machine-readable slot-zero audit hashes 27 official or upstream source receipts and evaluates a 576-value sentinel model. It made zero model calls, downloaded zero model-weight bytes, imported SGLang and AITER zero times, ran zero CPU or GPU inference jobs, started zero serving processes, ran zero containers, and applied zero third-party patches. The audit proves a pinned source boundary and a simple write predicate. Static source and a JavaScript memory model can prove the reserved-slot contract and current merge boundary; they cannot reproduce the GPU defect, validate the reporter’s measurements, attribute every repeated token to this cause, or certify a production deployment.

In this guide

1. Read the writer-specific verdict
2. Triage the symptom without assuming the cause
3. Trace the reserved slot contract
4. Separate the four writer paths
5. Check v0.5.18 against the merged SGLang fix
6. Audit the still-open fused AITER boundary
7. Run a deterministic sentinel gate
8. Capture the effective runtime identity
9. Build the full GLM-5.2 acceptance run
10. Mitigate without an unsafe hot patch
11. Rehearse on rented infrastructure carefully
12. Frequently asked questions
13. Bottom line
14. Sources and method

Read the writer-specific verdict

One version number cannot answer this incident. Four ownership boundaries have different states.

Runtime path	Pinned state on August 30, 2026	Decision
Generic SGLang KV store	PR #32477 merged and reported 340 focused passing tests	Contract established, but not MLA closure
SGLang-owned MLA BF16, FP8, scale and CUDA TMA/JIT writers	PR #36003 merged as 04c1036..., after v0.5.18	Hold until a pinned build and runtime gate pass
Fused AITER GLM ROCm writer	AITER PR #5010 open and unmerged	Reject for promotion
Future AITER API called from SGLang	No pinned downstream pad_slot_id=0 opt-in captured	Reject until end-to-end integration is pinned and tested

The practical verdict is therefore identify, then gate. If the deployment uses a SGLang-owned MLA writer, a build containing merge commit 04c1036bb395cce7d9b5eab1a814163ab1dcefed contains the source guard, but source presence is only the start of acceptance. If the GLM ROCm path writes KV inside AITER, SGLang PR #36003 is outside the writer and cannot protect it.

That distinction prevents two common but opposite mistakes. The first is remaining on v0.5.18 because its label appears in a vendor image; the release commit predates the standard-writer fix. The second is moving to an arbitrary SGLang main build and declaring the entire AMD path repaired; the merged PR explicitly says the fused writer bypasses it. “Newer” is not a writer identity.

This page owns that incident gate. The AMD ROCm deployment matrix owns hardware, image, checkpoint and topology selection. The HiCache guide owns L2/L3 cache offload. The MTP guide owns draft depth and acceptance. The DSA indexer fusion audit owns fused indexer stream ordering. Passing any one of those contracts does not preserve the physical padding slot.

Triage the symptom without assuming the cause

The upstream SGLang issue #36207 describes affected concurrent requests collapsing into short nonsensical text followed by one repeated token. In the reported sample, token ID 154822 repeats. That output is useful as an alarm, not a fingerprint. Repetition can also come from sampling settings, stop-token handling, a damaged weight shard, tokenizer mismatch, memory corruption elsewhere, a speculative state error, bad cache restore, or an application loop that replays output.

Use three evidence layers instead of naming the defect from text alone:

1. Output evidence: preserve request ID, prompt hash, token IDs, finish reason, reasoning/content separation, tool-call fragments and the first position at which repetition begins. Do not store sensitive prompt text if a hash and a synthetic reproduction are enough.
2. Tensor evidence: record finite checks at the writer boundary, slot 0 before and after the write, the source padding row, and the first downstream layer that becomes non-finite. A repeated-token detector without tensor evidence cannot isolate this path.
3. Identity evidence: record the installed SGLang and AITER commits, container digest, model revision, hardware, ROCm/CUDA version, KV dtype, graph mode, DP/DCP state, MTP state and selected writer. A tag copied from a launch script is not proof of the loaded code.

The incident report supplies all three kinds for one environment: eight AMD MI350X devices, ROCm 7.2.4, a dated v0.5.18-derived image, SGLang revision db570fe619ae15d8bd4ee04f146bc8234c52a20d, amd/GLM-5.2-MXFP4, BF16 KV, TP8, DP4 with DP attention, EP8, MTP, and DSA TileLang prefill/decode. The reporter counted 438 bad comparison events and 4,914 bad TileLang rows before a producer-side guard, then zero tensor anomalies and zero strong output failures in the comparable patched gate.

Those counts remain reporter observations. We did not run that hardware or download the model. Their value is that they connect the text symptom to a specific non-finite write and a controlled code change; they are not portable performance numbers or proof that another repeated response has the same root cause.

Trace the reserved slot contract

SGLang’s pinned memory_pool.py repeatedly names physical slot 0 as the padded or reserved slot. Padding rows exist so CUDA graphs, DP attention and related fixed-shape execution can keep a stable launch shape when not every rank has equivalent useful work. The row is not ordinary request state and must remain a finite safe source for later padded reads.

The dangerous sequence is short:

1. scheduling assigns a padding row to destination location 0;
2. values in that padding source are undefined and may contain NaNs;
3. an MLA-specific writer treats zero as an ordinary cache destination;
4. slot 0 loses its finite sentinel and becomes non-finite;
5. a later padded or masked attention row reads the reserved slot;
6. masking may not save the computation because zero multiplied by NaN is still NaN under IEEE floating-point rules;
7. the non-finite value reaches attention output, logits and finally malformed text.

The relevant GLM cache row is 576 values wide. The pinned amd/GLM-5.2-MXFP4 config declares a 512-dimensional compressed KV latent and a 64-dimensional RoPE component. It also declares GlmMoeDsaForCausalLM, index_topk=2048, one MTP layer, and shared index state for MTP iterations. The official zai-org/GLM-5.2-FP8 config cross-checks the architecture and DSA fields. Model metadata explains why the path matters; it does not prove that a particular runtime selects one writer.

The guard must satisfy two properties together. It must skip exactly the reserved destination, and it must preserve normal positive destinations. A fix that drops all writes when any row is padding prevents poisoning but also corrupts valid requests. That is why both slot 0 and a positive control slot belong in the same deterministic test.

Separate the four writer paths

The generic cache store first received a reserved-index contract in SGLang PR #32477. Its default skips index 0, with -1 as an explicit opt-out. The PR reports 340 focused tests. That work established the invariant, but MLA writers bypass the generic store and therefore did not inherit it automatically.

SGLang PR #36003 extends the contract to SGLang-owned MLA code. Its six changed files cover:

• the BF16 Triton MLA data writer;
• the FP8-quantized Triton MLA writer;
• the separate scale-buffer writer;
• the CUDA TMA/JIT bulk-store writer;
• focused kernel tests; and
• one prerequisite test call adjusted for the new parameter.

At the merge commit, the pinned mla_buffer.py defaults reserved_skip_index to zero and places the comparison inside the existing write predicate. The pinned set_mla_kv_buffer.cuh does the equivalent check for the CUDA writer. Launch shapes and positive-slot behavior remain unchanged, and -1 retains explicit legacy behavior.

The merged test_set_mla_kv_buffer.py includes BF16, int32/int64 location, FP8-quantized, scale-buffer and opt-out cases. Its sentinel assertions are much more useful than a launch-only test. The PR author also records an important limit: local GPU tests were not run; the focused cases were registered for CUDA and AMD CI.

The fourth path is different. SGLang’s pinned rocm_linear_utils.py imports the fused AITER MLA rope/cat/cache writer. That function writes the KV cache internally, so a later SGLang call can run attention with cache saving disabled. It bypasses every writer changed by PR #36003. Treat the import and runtime selection as separate questions: source availability does not prove that a deployment selected the fused branch, but selection makes the standard guard irrelevant.

Check v0.5.18 against the merged SGLang fix

The immutable v0.5.18 commit is dated August 20. The MLA writer fix merged on August 26. The v0.5.18 mla_buffer.py receipt contains no reserved_skip_index symbol; the merge snapshot contains the default and three guarded Triton write sites. That is a source fact, not a claim that every image labeled v0.5.18 is identical.

Use an immutable runtime manifest rather than the friendly tag:

Minimum slot-zero incident manifest

{
  "container_digest": "sha256:<record-the-pulled-image-digest>",
  "sglang_commit": "<40-hex-installed-source-commit>",
  "aiter_commit": "<40-hex-installed-source-commit>",
  "model_revision": "386bd0e4ec821f7b07975701cec3c3b953a5576a",
  "hardware": "8x exact accelerator and interconnect",
  "rocm_or_cuda": "<driver-and-runtime>",
  "kv_cache_dtype": "bfloat16",
  "parallelism": {"tp": 8, "dp": 4, "ep": 8},
  "dp_attention": true,
  "mtp": true,
  "mla_writer": "<resolved implementation>"
}

Do not infer the commit from an image name, a Python package version string or a build date. Record the image digest at pull time, resolve the installed source where possible, and hash the decisive files. If the package strips Git metadata, compare file hashes against the pinned upstream bytes. A locally modified wheel with the same version label is a different runtime.

For a standard writer, source containing the guard makes the build eligible for the deterministic probe, not for production. The PR’s tests were designed to run on accelerators, and the full model introduces scheduling, graph, parallelism and fused-kernel interactions beyond the focused writer unit.

Audit the still-open fused AITER boundary

ROCm/AITER PR #5010 proposes the missing library-side control. The wrapper adds an optional pad_slot_id with a default of -1. This is intentionally backward compatible: existing callers continue to treat slot 0 as writable. When a caller passes zero, the proposed Triton and gfx1250 Gluon paths skip that destination.

The pinned proposal head 5448910ddc44a1539b001bf648a8ac75b8f85e98 passes the value as a compile-time kernel argument. The fused_kv_cache.py kernel contains guards in both write branches. The test_fused_kv_cache.py proposal covers default slot-zero writes, zero reserved in the mixed/decode branch, zero reserved in the extra-prefill branch, and a valid positive destination.

That source is promising but not deployable evidence:

• the PR is open and unmerged;
• no approval was present in the captured review response;
• the author left GPU execution to AITER CI;
• the default is deliberately -1, not zero; and
• no pinned SGLang downstream call passing pad_slot_id=0 was captured.

Three events are therefore required, not one. The AITER API and kernels must merge; SGLang must adopt the exact API and pass zero on its reserved-slot path; then the integrated package must pass focused and full-model tests. A future AITER release containing the API but a downstream caller omitting the argument retains legacy behavior by design.

Run a deterministic sentinel gate

The smallest useful test has two rows and two destinations. Initialize physical slot 0 to a finite sentinel, create one padding source containing NaNs, create one finite control source, map them to locations 0 and 2, invoke the effective writer, and compare both destinations byte for byte. Run it separately for every writer and dtype the deployment can select.

Our zero-runtime calculator models the predicate with plain JavaScript arrays:

Original predicate model, not a GPU reproduction

for (const [sourceRow, destination] of writes) {
  if (destination === reservedSlot) continue;
  pool[destination] = sourceRow.slice();
}

The result makes both invariants visible:

Fixture	Slot 0 finite	Slot 0 NaN	Positive slot 2
Guard absent	0 / 576	576 / 576	all 512 latent and 64 RoPE values written
Guard active	576 / 576	0 / 576	all 512 latent and 64 RoPE values written

This table is an original evidence block, not a performance benchmark. It proves what the comparison should do. It cannot prove that a Triton, CUDA, Gluon or AITER kernel compiled the same predicate, received the expected location tensor, or ran in the live model.

For the real focused test, record at least:

• writer name and function identity;
• installed source hash and compiled package identity;
• device architecture and runtime version;
• BF16, FP8-data and scale-buffer cases that the deployment can reach;
• location dtype, because int32 and int64 both appear in upstream coverage;
• slot 0 before and after;
• positive control before and after;
• launch success and synchronization result; and
• whether an explicit opt-out was present.

Fail if slot 0 changes at all, not only if it becomes NaN. The sentinel protects an invariant, so an apparently finite replacement is still corruption. Fail if the positive control is not written. Fail if the test cannot prove which writer executed. A test that silently falls back to another implementation is not coverage.

Capture the effective runtime identity

Dynamic dispatch is where source audits often become false assurance. A repository can contain several correct implementations while the deployed configuration selects an older or fused path. Capture the selection after configuration resolution and immediately before the first model request.

The incident manifest should answer these questions:

Question	Required receipt	Why it matters
Which artifact ran?	Container digest and package/file hashes	Tags can move or hide local changes
Which model contract loaded?	Hub revision and config hash	Community quantizations can change code and fields
Which writer executed?	Resolved backend name plus source hash	Standard and fused guards have different states
Which padding features were active?	Graph mode, DP attention, DCP/CP, batch shape	Slot zero is reached through padding behavior
Which cache format ran?	KV dtype, layout and page settings	Separate writers and scale paths may be selected
Which optional states ran?	MTP/EAGLE, HiCache, disaggregation	They change scheduling and state lifetimes

Do not log secrets, user prompts, raw authorization headers, cookies or unnecessary full model responses. Use synthetic prompts and salted hashes for incident correlation. Keep request-level traces bounded, access controlled and dated. The debugging record should make the runtime reproducible without turning production data into a new security problem.

Also capture negative evidence. If the runtime does not expose a reliable writer identity, record that as a blocker. If an image does not include its source commit, do not fill the field from a nearby release tag. If a test passes only after disabling a feature, label the surviving topology rather than claiming the original configuration is fixed.

Build the full GLM-5.2 acceptance run

A sentinel test is necessary and intentionally narrow. The promotion suite must exercise the scheduling conditions that create padding and the application contracts that can be damaged after a non-finite value reaches logits.

Start with deterministic synthetic traffic:

1. one request on one active DP rank while peer ranks are idle;
2. uneven concurrent batches that force padding;
3. graph capture and replay at the exact production sizes;
4. short and long prefills followed by bounded decode;
5. repeated start, stop and restart cycles;
6. the exact TP, DP, EP and MTP topology; and
7. the selected prefill and decode writer pair.

Then validate model behavior:

• every monitored tensor remains finite;
• token IDs do not enter an unexplained repetition loop;
• reasoning and visible content stay in their expected fields;
• stop tokens terminate at the intended boundary;
• tool names and JSON arguments survive validation;
• long-context answer fixtures match a pinned reference within the chosen tolerance;
• concurrent results remain attributable to their requests;
• cancellation releases request and cache state;
• restart does not reuse poisoned cache data; and
• rollback restores the last known-good digest and reproduces its receipt.

Compare the candidate with the last known-good route over the same sanitized corpus. Do not require byte-identical natural-language output unless decoding is deterministic and the runtime contract promises it. Instead, define exact checks where possible—finite tensors, stop behavior, tool JSON, fixture facts, request ownership—and bounded semantic checks where variation is expected.

Monitor at least four rejection signals during a canary: non-finite tensor count, repeated-token run length, strong-output failure count, and divergence from the reference corpus. Add latency and throughput only after correctness. A faster path that silently corrupts one request is not an optimization.

Mitigate without an unsafe hot patch

The safest immediate response to a suspected production incident is to stop promotion and return to a pinned route that already passed the same acceptance corpus. Do not apply an open pull request directly to a shared production image and infer success from server startup. A source patch can compile while the wrong writer still runs, the downstream call omits the new argument, or another branch remains unguarded.

Use this decision sequence:

1. quarantine the affected digest and preserve its minimal sanitized receipts;
2. verify whether the standard or fused writer executed;
3. if standard, build a task-owned candidate from a pinned commit containing 04c1036... and run the focused plus full suite;
4. if fused AITER, keep the production gate closed while the library proposal and downstream opt-in remain unmerged or unproven;
5. if disabling the fused path is a documented supported configuration, treat the resulting standard-writer route as a new candidate and rerun every gate;
6. otherwise roll back or route traffic to a separately validated API or accelerator path.

“Disable DP attention” or “disable graphs” may remove one trigger in a specific test, but neither is a general proof that the reserved slot cannot be reached. Those changes also alter performance and scheduling. Record them as distinct topologies, not as a universal fix. Similarly, clearing a cache after a bad response treats state, not the producer bug.

Never edit installed site-packages in place. Build an immutable candidate, record its patch and source hashes, and keep the last known-good artifact. If the candidate cannot be reproduced from the manifest or rolled back in one bounded step, it is not ready for traffic.

Rehearse on rented infrastructure carefully

After identifying a supported writer path, rented isolated GPUs can help rehearse the client, evaluator, finite checks, rollback and cleanup. Inventory must be verified live. A substituted NVIDIA GPU cannot validate the reported AMD MI350X, ROCm, AITER or TileLang behavior, and a single GPU cannot validate the reported TP8/DP4/EP8 topology.

Rehearse only the parts the available GPU can reproduce

Use a rental only after writing the exact experiment boundary. A different accelerator can rehearse request capture and the acceptance harness; it cannot close an AMD fused-kernel incident. Destroy only the task-owned instance and storage after preserving sanitized receipts.

Check current isolated GPU availability on RunPod →

Affiliate disclosure: This is an affiliate link and a referral link. GLM52.ai may receive a commission or referral credit after qualifying usage at no added charge from us. RunPod availability does not prove that AMD MI350X hardware, eight-device topology, ROCm 7.2.4, the affected AITER writer, sufficient memory, correct interconnect, storage, price, security, output correctness, or the slot-zero fix is available. Verify the live SKU, immutable image and source pins, model revision, test scope, terms, budget, cleanup and rollback before use.

If exact hardware is unavailable, do not invent an equivalence. A smaller or different device can validate the JavaScript policy, request corpus and result parser. It cannot validate the 576-value AITER writer, DP-rank padding, interconnect behavior or performance. Label every result by what the hardware actually covered.

Frequently asked questions

Does every repeated GLM-5.2 token mean slot 0 is poisoned?

No. Repetition is a symptom shared by sampling, tokenizer, stop handling, checkpoint, cache, speculation and application failures. Confirm the exact MLA writer, non-finite transition, slot-zero mutation and positive-slot control before assigning this root cause.

Is SGLang v0.5.18 protected?

Not by the merged standard-writer change audited here. The v0.5.18 commit predates PR #36003, and its pinned MLA source lacks reserved_skip_index. Vendor images can contain downstream patches, so inspect the actual file and digest instead of trusting or rejecting the label alone.

Does SGLang main fix GLM-5.2 on AMD?

That statement is too broad. The merged change protects SGLang-owned MLA writers. Its own scope excludes the fused AITER GLM writer. Determine whether the deployment reaches the changed functions before using the merge as evidence.

Can I apply AITER PR #5010 now?

Not as a production conclusion. The captured PR is open and unmerged, and the new argument defaults to -1. A downstream caller must explicitly pass zero. A controlled task-owned experiment can evaluate an open proposal, but it must not be mislabeled as released support.

Why does the proposal default to minus one?

AITER is a general library and cannot assume that every caller reserves slot 0. The proposed opt-in keeps existing callers compatible. That is why a future library merge alone is insufficient for SGLang: the affected integration must select pad_slot_id=0.

Is a finite slot 0 enough to pass?

No. It is the first invariant. The positive control must also write, all reachable writer and dtype variants must pass, and the complete GLM-5.2 runtime must clear concurrency, graph, reasoning, tools, long-context, restart and rollback gates.

Can disabling MTP, DP attention or CUDA graphs be the fix?

It can define a different candidate topology and may avoid a specific trigger. It does not prove that the original topology is safe or that no other padding path reaches slot 0. Benchmark and validate the changed topology separately.

Did this audit reproduce the GPU failure?

No. It fetched and hashed 27 public source artifacts and ran a deterministic JavaScript array model. It made no model call, imported no runtime, executed no GPU kernel and applied no patch. GPU counts and before/after runtime results remain attributed to the upstream reporter.

Bottom line

Treat GLM-5.2 repeated tokens as an alarm, then identify the writer. Physical slot 0 is reserved padding state; a NaN padding row must not overwrite it, and a valid positive destination must still be written. That two-part sentinel is the smallest reliable gate.

The pinned source state is asymmetric. SGLang PR #36003 merged the guard for its own MLA writers after v0.5.18. The fused AITER GLM ROCm route bypasses those functions. AITER PR #5010 proposes an opt-in guard but remains open, defaults to legacy behavior, and still needs a pinned SGLang caller plus integrated runtime evidence.

Do not promote from a version label, a server start or one clean response. Pin the image, source, model and topology; prove the effective writer; pass the sentinel; then pass the complete GLM-5.2 application and rollback suite. Until the fused path meets that end-to-end contract, hold it or use a separately validated route.

Sources and method

This article was checked on August 30, 2026. The public evidence JSON contains URLs, final URLs, HTTP status, byte count and SHA-256 for 27 receipts, plus the deterministic sentinel results and five fail-closed deployment fixtures.

Primary sources include the SGLang incident, merged MLA writer fix, earlier generic-store fix, open AITER proposal, pinned SGLang and AITER implementation and test files, the immutable AMD MXFP4 model config, the official Z.AI FP8 config, the GLM-5.2 release page, and the Zhipu AI research index. The last two are mandatory discovery checks, not runtime-fix evidence.

The source audit establishes what the captured revisions say and which files a merge changed. The original JavaScript model establishes only the logical effect of skipping destination zero over a 512 + 64 row. Neither evidence class validates compiled GPU code, live dispatch, the reporter’s measurements, model quality, performance, or future project state. Recheck the upstream PRs, the exact installed bytes and the full acceptance corpus before changing the decision.

About the editorial role

Evan Brooks

AI Infrastructure Editor

Evan Brooks serves as GLM52.ai’s disclosed editorial pen name for infrastructure, deployment, and training guides. The role examines checkpoint size, memory math, provider limits, implementation evidence, and operational risk. The GLM52.ai Editorial Team produces and fact-checks every article under this byline.

The GLM52.ai Editorial Team accepts responsibility for this article. Read our editorial policy or request a correction.

An AI image tool created this portrait. It does not depict a real individual.