GLM-5.2 SGLang MoE Bias: Audit FP32 Expert Routing

Evan Brooks

AI Infrastructure Editor

Editorial responsibility: GLM52.ai Editorial Team · Published and editorially reviewed August 31, 2026

How we research and review · Report a correction

Disclosed editorial pen name · AI-generated portrait

The proposed fix is open, unmerged and not in v0.5.18

SGLang PR #37133 changes both the GLM-5.2 correction-bias parameter dtype and the AITER top-k call boundary. At the pinned state checked on August 31, 2026, it has no approval, is not merged, is absent from the latest v0.5.18 release, and its three named aggregate workflows are red. Treat it as a proposal to evaluate, not a production patch or an upgrade recommendation.

Original editorial illustration. It is not a GLM-5.2 activation map, SGLang screenshot, checkpoint plot, benchmark, model response or proof that a running server contains the proposed fix.

Short answer: the official GLM-5.2 FP8 checkpoint stores every routed-layer e_score_correction_bias tensor in FP32, but the pinned SGLang base can downcast it twice: while constructing MoEGate for selected quantized AITER paths, then again immediately before the AITER grouped top-k call. Our exact range audit found 76 F32 tensors from layers 3 through 78. Every one loses distinct values under deterministic BF16 rounding: the median falls from 214 levels to 7. Keeping only the first site in FP32 is therefore insufficient.

The machine-readable checkpoint audit hashes 22 official or upstream source receipts and reads exactly 77,824 model- weight bytes—76 tensors × 256 values × 4 bytes—from the immutable official checkpoint. It downloads no complete shard, archives no raw tensor values, makes zero model calls, imports neither SGLang nor Torch, starts no server or container, runs no CPU or GPU inference, and applies no third-party patch. It proves a checkpoint precision pattern and a pinned source boundary. It does not prove runtime correctness or reproduce the PR author’s routing, accuracy or latency measurements.

In this guide

1. Read the current verdict
2. Understand what the correction bias controls
3. Audit all 76 official tensors
4. Resolve the layer 3 and layer 10 numbers
5. Trace the first downcast at weight loading
6. Trace the second downcast at the AITER boundary
7. Read the open proposal without overclaiming
8. Verify the loaded runtime instead of the file
9. Build an expert-ID acceptance canary
10. Use a fail-closed promotion matrix
11. Interpret accuracy and latency evidence carefully
12. Rehearse on isolated hardware
13. Frequently asked questions
14. Bottom line
15. Sources and method

Read the current verdict

There are three different statements that are easy to collapse into one:

Question	Pinned answer on August 31, 2026	Operational decision
Does the official checkpoint store the routed correction biases in FP32?	Yes: 76/76 descriptors are F32 [256]	Necessary evidence, not runtime proof
Does the captured SGLang base preserve those values through the affected AITER path?	No: one branch allocates BF16 and another converts to the gating dtype	Reject that path for a fidelity-sensitive promotion
Is the upstream two-site fix ready to deploy?	No: PR #37133 is open, unmerged, unapproved and outside v0.5.18	Hold; test a future pinned merge and release

The safe decision is not “BF16 is always bad” or “apply this PR.” It is pin, inspect, then canary. Pin the checkpoint, runtime commit, quantization config, architecture string and top-k backend. Inspect the loaded parameter dtype and the actual AITER call inputs. Only then decide whether the captured failure mode applies.

This page owns correction-bias precision and expert-ID fidelity. The expert-parallelism guide owns placement, all-to-all communication and capacity. The DSA indexer fusion audit owns indexer stream ordering. The KV slot 0 audit owns MLA cache-write corruption. The AMD SGLang deployment matrix owns hardware, checkpoint and topology selection. A deployment can pass three of those contracts and still load the wrong correction-bias dtype.

Understand what the correction bias controls

GLM-5.2 is a sparse mixture-of-experts model. The pinned official configuration declares 256 routed experts and chooses eight for a token. Its topk_method is noaux_tc: in the relevant grouped router, the selection score combines a sigmoid-transformed gating logit with the learned correction bias. A simplified ordering expression is:

selection_score[token, expert]
  = sigmoid(gating_logit[token, expert])
  + e_score_correction_bias[expert]

The bias does not merely scale the final expert output. It participates in which experts enter top eight. A conversion that preserves the broad magnitude but collapses nearby values can therefore change ordering. That is a routing- fidelity issue even when the resulting text still looks plausible.

BF16 keeps an eight-bit exponent but only seven explicit fraction bits. Its spacing grows with magnitude. Near 7, adjacent representable values are 0.03125 apart; near 34, they are 0.25 apart. A band only a few tenths wide can contain hundreds of distinct FP32 values but only a handful of BF16 levels. The problem is not that 7 or 34 overflows. The problem is resolution within a narrow, offset band.

This distinction also explains why generic advice such as “the model is FP8, so BF16 auxiliary tensors are already high precision” is too coarse. FP8 is the primary weight quantization format. The official config deliberately lists correction-bias tensors among modules not converted. Auxiliary precision is a per-tensor contract, not a label inherited from the checkpoint name.

Audit all 76 official tensors

We pinned zai-org/GLM-5.2-FP8 at revision ba978f7d347eaf65d22f1a86833408afdb953541. Its model index maps 118,629 tensors across 755,617,140,416 bytes. We selected only names matching model.layers.N.mlp.gate.e_score_correction_bias.

The 76 matches cover layer 3 through layer 78. That fits the pinned config: three initial dense layers, routed MoE layers thereafter, 78 target layers and one NextN layer. Every descriptor says F32, shape [256], for 1,024 bytes.

For each shard, the collector requested its eight-byte safetensors length prefix, exact JSON header, then the target tensor’s exact byte interval. It required HTTP 206 and matching Content-Range, hashed the header and tensor, decoded 256 little-endian floats, calculated aggregates, and discarded the values. No request downloaded a complete multi-gigabyte shard.

Checkpoint-wide measure	Audited result
Routed correction-bias tensors	76
Layers covered	3–78
Values per tensor	256
Exact model-weight bytes read	77,824
Equivalent BF16 storage	38,912 bytes
Additional FP32 storage	38 KiB
Median FP32 distinct values per layer	214
Median BF16 distinct values per layer	7
Minimum / maximum BF16 levels	3 / 16
Layers losing distinct levels	76 / 76
Layers changing top eight with all logits fixed at zero	75 / 76

That 38 KiB delta is tiny beside a 755.6 GB tensor index. It does not prove the runtime cost is zero—the router may promote logits and use a different kernel path—but it does show that checkpoint storage or parameter memory is not a credible reason to destroy the bias precision.

The zero-logit top-eight comparison is deliberately limited. It ranks the actual bias values before and after BF16 conversion while holding every logit at zero. It is a sensitivity probe, not a traffic estimate. Layer 16 is the only layer whose set is unchanged in that artificial condition; its 216 FP32 levels still collapse to seven. Real token logits can change ties and group selection in either direction.

Resolve the layer 3 and layer 10 numbers

The upstream material initially looks inconsistent. The PR body says the bias sits near 7 and reports 238 distinct FP32 values becoming eight. The new test file describes a synthetic range near 34 with 174 values becoming roughly three. The checkpoint-wide audit shows both descriptions correspond to real, different layers.

Layer	FP32 min–max	Spread	FP32 levels	BF16 levels	Zero-logit top-8 members replaced
3	33.985321–34.622971	0.637650	174	3	8 / 8
10	6.817389–7.063244	0.245855	238	8	4 / 8
30	3.688965–4.033890	0.344925	235	16	8 / 8
47	6.058792–6.645473	0.586680	220	15	7 / 8
78	15.421862–15.817717	0.395855	240	7	8 / 8

Layer 10 matches the PR body after rounding: its exact 256-value tensor has SHA-256 907a7863bd51c582a81dea96318f5bebf4cd87cff1239d8e43d489705bf08804. Layer 3 matches the test’s mechanism: the actual checkpoint has 174 distinct FP32 values and three BF16 values near 34. The test does not load that tensor; it creates 174 linearly spaced synthetic values from 34 across a half-point window. Still, the correspondence explains why its comment differs from the body.

This is the principal new evidence in our audit. It independently verifies the checkpoint distributions behind the two explanations without claiming to verify the author’s real-logit route comparison. The author reports 98.50% of tokens choosing a different top-eight set. We did not receive those logits or run the probe, so that percentage remains attributed rather than reproduced.

Our own JavaScript fixture also uses synthetic values. With a fixed seed, 174 linearly spaced values near 34 become three BF16 levels; 60 of 64 synthetic tokens change their top-eight set, or 93.75%. That demonstrates the mechanism. It is not an estimate of production frequency and must not be compared as if it were a second measurement of 98.50%.

Trace the first downcast at weight loading

At SGLang base revision 26c754e06ea60c2293098a1faa75f622c8256ab0, MoEGate.__init__ starts correction_bias_dtype as FP32. When a quantization configuration is present and the selected AITER quantizer is FP8, compressed-tensors or Quark, the branch changes the allocation to BF16. Weight loading then casts the official FP32 checkpoint values into that already-BF16 parameter. Once rounded there, promoting the parameter later cannot recover the lost ordering.

The proposal adds _is_glm_moe_dsa(config) and prevents that downcast for an architecture string containing GlmMoeDsa. This includes the target GlmMoeDsaForCausalLM and the draft architecture GlmMoeDsaForCausalLMNextN. The pinned model_config.py explicitly rewrites the main name to the NextN name for a draft model, so testing both is material, not decorative.

The proposal’s blast-radius guard is also important. A non-GLM AITER FP8 configuration is still expected to allocate BF16. Broadly forcing every MoE correction bias to FP32 could change unsupported backends, memory layouts or kernel expectations. An architecture gate is safer than a universal dtype override, but only after maintainers validate that identification contract.

Trace the second downcast at the AITER boundary

Fixing MoEGate alone leaves another conversion. In the base biased_grouped_topk_gpu, the AITER branch creates bias by converting the correction parameter to gating_output.dtype. If gating output is BF16, an FP32 parameter is rounded again just before routing.

PR #37133 proposes a narrow split. If correction_bias.dtype is already FP32, it keeps that tensor and promotes gating_output to FP32 for the AITER call. Otherwise it sends the existing gating output and converted bias exactly as before. The branch keys on the bias dtype, not the architecture string.

That choice creates a useful two-part invariant:

loaded correction bias is FP32
AND
the selected AITER grouped-top-k call receives FP32 bias + FP32 gating logits

Both must be observed. If the first is false, the checkpoint information was destroyed at load time. If the second is false, the correct parameter is destroyed at dispatch time. A unit test that asserts only gate.e_score_correction_bias.dtype cannot close the full path.

The PR author notes that the same local bias also feeds a Kimi-K3 radix-4 path earlier in the function. The proposed change leaves that route unchanged and branches only at the AITER call. This page does not evaluate Kimi behavior, and no MiniMax model or code was used in this work.

Read the open proposal without overclaiming

The captured PR changes three files with 164 additions and three deletions. Its new 141-line test module registers nine CPU tests. They cover main and NextN GLM architecture matching, a non-GLM BF16 control, empty architecture lists, synthetic BF16 level collapse and synthetic top-eight divergence.

Those are well-targeted tests on the proposed contract. They are not evidence that the current PR passed. At collection time, the Base, Extra and AMD ROCm 7.2 aggregate workflows all concluded failure, while many child jobs were shown as skipped. The visible state does not establish whether the focused CPU tests actually ran and failed, whether repository policy prevented them from running, or whether an unrelated gate failed. Therefore the correct label is red and unresolved, not “the fix is broken” and not “nine tests pass.”

The latest upstream release API resolved to SGLang v0.5.18, published August 22, at commit 71de97b264b04dcd514cf904003028aefe9775c8. The proposal was opened August 30 and is not included. A package named v0.5.18 cannot contain this PR unless a vendor has applied an untracked patch—which would make the package identity more, not less, important.

Do not copy the two hunks into a production image merely because the numerical mechanism is convincing. An open patch may change during review; architecture dispatch, compilation, device behavior and kernel contracts need maintainer and CI evidence. A private test build can evaluate the proposal within an isolated, reversible canary. It must not be presented as an upstream release.

Verify the loaded runtime instead of the file

A Hugging Face descriptor proves the saved bytes, not the live parameter. Add a startup assertion to a task-owned test build after weight loading and before serving traffic. Adapt the module path to the exact SGLang revision rather than copying a fragile object traversal from another version.

audited = []
for name, parameter in model.named_parameters():
    if name.endswith("mlp.gate.e_score_correction_bias"):
        audited.append((name, parameter.dtype, parameter.numel()))

assert len(audited) == 76, audited
assert all(dtype == torch.float32 for _, dtype, _ in audited), audited
assert all(count == 256 for _, _, count in audited), audited

Record the assertion beside these immutable or effective identities:

1. checkpoint repository and revision;
2. model index hash and all loaded shard hashes or verified manifests;
3. SGLang commit, not only package version;
4. AITER commit and wheel/container digest;
5. quantization configuration name and loaded architecture strings;
6. target and NextN layer counts;
7. selected top-k function and device backend;
8. GPU model, driver, CUDA or ROCm version;
9. TP, DP, EP, MTP and graph settings; and
10. the sanitized test corpus and evaluator revision.

Instrument the AITER boundary in the same isolated build. Assert the bias and gating tensors are FP32 on the exact call taken by representative requests. Remove diagnostic logging before production; logging entire biases, logits or prompts is unnecessary and can create data exposure. Dtypes, shapes, hashes, counts and aggregate expert-ID differences are sufficient receipts.

Fail closed on a count other than 76 for this pinned checkpoint. A different future checkpoint may legitimately change the layer structure, but silently accepting a different count means the audit no longer describes the runtime. Update the expected contract only after pinning and reviewing that model.

Build an expert-ID acceptance canary

Parameter dtype is necessary but still not output proof. The acceptance canary should compare the same token-level gating inputs through a control and candidate route while preserving grouping, top-k and scaling behavior. Capture expert IDs and weights, not only final text.

Use at least these workload classes:

• short factual and coding prompts that exercise ordinary decode;
• long prefill followed by enough decode to sample many layers and tokens;
• reasoning enabled and disabled where supported;
• structured tool calls with schema validation;
• MTP enabled and disabled, because the NextN architecture is explicitly in the proposal;
• concurrency and batch shapes that select the production AITER backend;
• deterministic seeds and greedy controls where the serving API allows them;
• cancellation, restart and cache-warm transitions.

For every routed token/layer pair, compare the unordered top-eight expert set first. Then compare ordering, normalized weights and downstream hidden-state statistics. A set change is stronger evidence of a routing decision change than a small weight delta. Yet it is not automatically an output failure: experts may be redundant and later layers can absorb differences.

Keep three result layers separate:

Evidence layer	Question	Example gate
Routing fidelity	Did the candidate preserve intended expert IDs from FP32 bias?	Expected set match on a trusted FP32 reference route
Output correctness	Did responses remain valid for the reader’s tasks?	Tool-schema pass, answer checks, no non-finite tensors
Service cost	Is the precision path affordable at target load?	p50/p95 TTFT, TPOT, throughput, memory and queue limits

The trusted reference cannot be “whatever the old server returned” because the old path may be the affected path. Use an explicit FP32 router reference whose input logits, group configuration and bias are pinned. If two device kernels are expected to differ in floating-point reductions, define tolerances for weights while keeping exact or justified set-level requirements.

Use a fail-closed promotion matrix

The machine-readable calculator evaluates five illustrative states:

Candidate state	Parameter site	AITER site	Merge/release	Runtime canary	Decision
Captured quantized base path	BF16	BF16	Not fixed	Missing	Reject
Parameter-only hot patch	FP32	BF16	Private partial patch	Missing	Reject
Both hunks from open PR	FP32	FP32	Open, unmerged	Missing	Hold
Future pinned merge, source tests only	FP32	FP32	Included	Incomplete	Hold
Future pinned release with complete acceptance	FP32	FP32	Included	Passed	Eligible for bounded canary

“Eligible” is not “safe forever.” Start with bounded traffic, keep a known rollback image and watch expert-set differences, non-finite tensors, output validation, latency and memory. Stop the canary if any identity changes or if the selected backend differs from preflight. A deployment restart can choose a different graph, kernel or package artifact even when the command line looks the same.

Rollback should restore a known service state, but it does not make the old routing more faithful. If the current production build demonstrably destroys the precision contract, route affected traffic to a verified alternative backend, disable the affected quantized AITER route if supported, or hold the deployment. Do not call rollback to a known-bad path a correctness fix.

Interpret accuracy and latency evidence carefully

The PR author reports GSM8K moving from 0.941 to 0.947 and GPQA-Diamond from 0.8333 to 0.8182, describing both as within error bars. We did not run those benchmarks, inspect their item-level outputs or verify the error calculation. The numbers cannot support a claim that FP32 improves accuracy, harms accuracy, or has no task effect.

This is exactly why endpoint accuracy is an incomplete routing test. Sparse experts can overlap in capability. A route can differ on most tokens while a small benchmark score stays similar. Conversely, faithful expert selection does not guarantee every downstream answer improves. Use the accuracy suite to bound task regressions and the expert-ID probe to test routing fidelity.

The author also reports about 4.5 microseconds per gating call and at most 1.2% of a decode step. Those values are environment-specific and not reproduced here. Promoting gating logits to FP32 can affect memory traffic, kernel choice and device utilization differently across GPUs, batch sizes and graph modes. Measure the exact production topology with confidence intervals or repeated runs. Do not multiply one microbenchmark by layer count and present it as an end-to-end forecast without accounting for overlap and launch behavior.

The 38 KiB parameter-memory delta is independently derived and exact for these 76 tensors. It does not include temporary FP32 gating logits. For a token batch of T, that temporary logical payload is T × 256 × 4 bytes before kernel- specific allocation and reuse. Capture peak allocated and reserved memory instead of assuming the tensor is materialized exactly once.

Rehearse on isolated hardware

After a merge is pinned and the source-level gates are green, isolated rental hardware can help run the dtype assertion, expert-ID canary, output suite, latency sweep, rollback and cleanup. Verify the live GPU SKU, count, interconnect, image, storage, region, price and availability before provisioning. A different accelerator can validate the harness but cannot establish the production AITER or ROCm/CUDA kernel result.

Rehearse the FP32 routing canary on isolated GPUs

Provision only after the proposal is merged into a pinned test build and the two dtype assertions are written. Use a different GPU only for harness rehearsal; use the exact production backend for kernel and performance conclusions. Preserve sanitized receipts, then delete only the task-owned instance and storage.

Check current isolated GPU availability on RunPod →

Affiliate disclosure: This is an affiliate link and a referral link. GLM52.ai may receive a commission or referral credit after qualifying usage at no added charge from us. RunPod availability does not prove that your required GPU, topology, SGLang or AITER commit, GLM-5.2 checkpoint, interconnect, storage, price, security boundary, FP32 routing behavior, output quality or latency target is available. Verify every live property, budget, cleanup and rollback condition before use.

Do not download hundreds of gigabytes merely to check this issue. The official index identifies the 76 tiny tensors, and an exact range audit can validate the saved values before renting a GPU. GPU time becomes useful only when testing loaded dtype, selected kernel, expert IDs, end-to-end output and service cost.

Frequently asked questions

Is the official GLM-5.2 FP8 checkpoint incorrectly quantized?

No evidence here supports that conclusion. The pinned checkpoint stores all 76 correction-bias tensors as F32 and lists those names among modules not converted. The captured loss occurs in SGLang runtime dtype choices for a specific quantized AITER path, not in the saved tensor descriptors.

Does every SGLang backend downcast the bias?

No. The finding is scoped to the pinned branches inspected for PR #37133. SGLang contains multiple grouped top-k implementations and device paths. Verify the function actually selected by the deployment. Do not generalize an AITER branch to FlashInfer, Triton, CPU or a future implementation without source and runtime evidence.

Can I fix the issue by casting the parameter after loading?

No. Casting a BF16 parameter back to FP32 preserves the already-rounded BF16 values; it cannot reconstruct the original distinct levels. The parameter must be allocated or loaded in FP32 from the checkpoint, and the AITER boundary must avoid a second downcast.

Why does the article discuss 76 tensors when the config says 78 layers?

The first three layers are dense rather than routed MoE layers, and the pinned model also includes one NextN layer. The routed correction-bias names span layers 3 through 78 inclusive, which is 76 tensors. The exact index mapping is the authoritative count for this revision.

Does 75 of 76 changed top-eight sets mean 98.68% of tokens change route?

No. The 75/76 result counts layers under an artificial all-zero-logit test. It does not count tokens and cannot be converted into a production token rate. The PR author’s 98.50% is a separate, author-reported real-gating-output probe that this audit did not reproduce.

Do the red workflow badges prove the proposed code is wrong?

No. They prove the captured aggregate runs concluded failure. Many child jobs were skipped, and the public state inspected here does not establish that the nine focused tests ran and failed. The unresolved red state blocks a confident promotion, but its cause needs maintainer or job-level evidence.

Should I disable BF16 everywhere in SGLang?

No. BF16 remains appropriate for many tensors and kernels. The problem is the precision requirement of this narrow, offset correction-bias vector and the two affected conversion sites. Apply the smallest verified scope and preserve non-GLM controls.

Bottom line

The official checkpoint evidence is clear: 76 GLM-5.2 routed correction-bias tensors are F32, and all 76 lose distinct levels under BF16 conversion. Layer 10 independently reproduces the PR body’s 238 → 8 checkpoint distribution; layer 3 explains the test’s 174 → 3 description. The storage cost of keeping all of them FP32 is only 38 KiB above BF16.

The deployment answer is less final. SGLang PR #37133 correctly identifies two separate precision boundaries, but it is an open, red, unmerged proposal that is not part of v0.5.18. Keep affected promotions on hold. Require both loaded parameter and AITER-call dtypes to be FP32, pin a future merged build, run the focused tests, compare expert IDs and output tasks, measure service cost, and then use bounded traffic with a rehearsed rollback.

Sources and method

Primary sources include the SGLang proposal, its pinned base and head, the official GLM-5.2 FP8 checkpoint, the SGLang v0.5.18 release, Z.AI’s GLM-5.2 release page, and the Zhipu AI research index. The latter two were mandatory discovery checks, not evidence for SGLang kernel behavior.

The AI HOT batch supplied a MiniMax H3 Max item, a Hugging Face breach item, and a ChatGPT Work item. They were respectively classified as weak/prohibited, duplicate and weak, so proactive first-party research supplied this topic. One exact Google query via SerpAPI failed without a retry and did not influence the decision.

The public evidence JSON contains source hashes, immutable pins, every tensor’s HTTP range and SHA-256, per-layer aggregates, the deterministic synthetic fixture and five fail-closed deployment states. Raw tensor values are not retained. Check date: August 31, 2026. Recheck the PR, release, workflow and runtime state before acting.

About the editorial role

Evan Brooks

AI Infrastructure Editor

Evan Brooks serves as GLM52.ai’s disclosed editorial pen name for infrastructure, deployment, and training guides. The role examines checkpoint size, memory math, provider limits, implementation evidence, and operational risk. The GLM52.ai Editorial Team produces and fact-checks every article under this byline.

The GLM52.ai Editorial Team accepts responsibility for this article. Read our editorial policy or request a correction.

An AI image tool created this portrait. It does not depict a real individual.