Skip to content

GLM-5.2 vLLM Sequence-Parallel MoE Rollout Audit

Independent research — not an official Z.ai publication.Identity and provider disclosure

A cool-blue topology diagram shows two data-parallel lanes with four tensor-parallel nodes each feeding a shared MoE expert fabric; the lower low-load path reaches a small amber and red mismatch gate, while a separate teal rollback route carries state safely around it

Original editorial illustration: cyan ribbons represent sharded token state, the small amber gate marks the one-token guard collision, and the teal route is a predeclared rollback. It explains the audit boundary; it is not a vLLM trace or performance result.

Sequence-parallel MoE is an internal communication optimization, not a flag a GLM-5.2 operator necessarily typed. In vLLM, the runtime derives it from the all-to-all backend and the TP, DP, EP, and PP layout. That makes a source audit important: two deployments can use the same checkpoint and both say “expert parallel,” while only one carries a local token shard between MoE layers and tries to infer that state from tensor shape.

The failure mode is unusually dangerous because the server can stay up and return HTTP 200 while text quality collapses. A health check that opens a port, loads weights, or generates any nonempty string does not protect the service. Low traffic can be the sharper canary than high traffic because a single active decode token on one DP rank is the exact arithmetic collision reproduced from the pinned source.

This guide pins vLLM v0.27.1 at commit 6e448d0ea9bf3d88d898b65449ca6dc2aec170ac and GLM-5.2-FP8 revision ba978f7d347eaf65d22f1a86833408afdb953541. The machine-readable receipt hashes twenty-seven public artifacts and enumerates forty-eight shape cells. We made zero model calls, zero GPU runs, and zero local upstream-test runs. Reporter observations remain reporter evidence; source arithmetic remains source arithmetic.

  1. Decide whether the runtime enters SP-MoE
  2. Follow state through one decoder layer
  3. Reproduce the one-token guard collision
  4. Separate the proven boundary from the issue claim
  5. Map stable releases and the nightly window
  6. Audit the open fix before trusting it
  7. Pin runtime provenance without importing CUDA
  8. Build a low-load content canary
  9. Grade output rather than process health
  10. Keep correctness and performance gates separate
  11. Choose a reversible control
  12. Rent only after the canary contract exists
  13. Troubleshoot ambiguous results
  14. Frequently asked questions
  15. Sources and method

At the pinned v0.27.1 source, ParallelConfig.use_sequence_parallel_moe becomes true only when four conditions hold:

Pinned vLLM activation predicate
eligible all-to-all backend
AND expert parallelism enabled
AND tensor parallel size > 1
AND data parallel size > 1

The default allgather_reducescatter backend is eligible. So are the pinned DeepEP high-throughput and low-latency choices, both MoRI choices, and nixl_ep. Changing from the default to one of those does not remove the sequence-parallel state detector.

The GLM/DeepSeek decoder layer adds two more boundaries: the layer must be an MoE layer, and pipeline parallel size must equal one. GLM-5.2’s pinned config reports model type glm_moe_dsa, 78 decoder layers, 256 routed experts, and eight selected experts per token. Those architecture facts establish that the MoE branch exists; the parallel layout decides whether this particular state path is active.

The reproduced topology matrix is:

Topology at the audited stable source Global SP-MoE property Decoder-layer path Decision
TP4, DP1, EP on, default backend, PP1 Off Off Not exposed to this exact DP-dependent path
TP4, DP2, EP on, default backend, PP1 On On Low-load output canary required
TP8, DP2, EP on, DeepEP low latency, PP1 On On Backend choice does not remove the guard
TP1, DP2, EP on, default backend, PP1 Off Off TP precondition absent
TP4, DP2, EP off, default backend, PP1 Off Off EP precondition absent
TP4, DP2, EP on, default backend, PP2 On Off Layer-level PP boundary blocks this exact path
TP4, DP2, EP on, naive, PP1 Off Off Different communication implementation; not proof of safety

This table is a source-path classifier, not a recommendation to choose a slow or unsuitable topology. DP1 duplicates no model replica inside the audited SP-MoE property, while DP2 can raise capacity and scheduling complexity. PP2 changes layer partitioning. naive changes expert exchange. Each alternative has its own memory, communication, availability, and performance consequences.

For rank, slot, and redundant-expert sizing, use the separate GLM-5.2 expert-parallelism guide. That page answers “how should experts be placed and balanced?” This one answers “does the chosen vLLM graph preserve low-load output before it is promoted?”

The pinned deepseek_v2.py captures the full local token count from positions.shape[0]. If the input is recognized as already sequence parallel, it all-gathers the local hidden rows before attention and trims padding back to the full token count.

After attention, an SP-MoE layer does the reverse:

  1. Pad the token axis to a multiple of TP.
  2. Reduce-scatter the hidden state across TP ranks.
  3. Chunk the residual to the same local length when the incoming state was not already recognized as sequence parallel.
  4. Run the local MoE with already_sequence_parallel=True.
  5. Carry local hidden and residual state into the next decoder layer.

The pinned sequence_parallel_chunk uses the same geometry: pad the sequence length to divisibility, then take one equal chunk for the TP rank. If full token count is n and TP size is p, the local row count after either operation is therefore:

local_rows = ceil(n / p)

The next layer’s old detector does not carry an explicit state bit. It infers state with a shape comparison:

Old carry-state guard in vLLM v0.27.1
input_is_sequence_parallel = (
self.use_sequence_parallel_moe
and residual is not None
and hidden_states.shape[0] != full_num_tokens
)

Shape is normally an efficient signal. It fails when the scattered local row count and the original full count happen to be equal.

Substitute the source geometry into the guard. The shape comparison is false when:

ceil(n / p) = n

For positive integer n and p > 1, that equality holds only at n = 1. Once n is at least two, dividing by a TP size of at least two and rounding up still produces fewer than n rows.

The deterministic audit enumerated token counts 1 through 16:

TP size Proven old-guard collision Below-TP counts the old guard already detects Counts changed by PR #50155 fallback
2 1 none 1
4 1 2, 3 1
8 1 2, 3, 4, 5, 6, 7 1

At TP4 and n=1, padding produces four rows and reduce-scatter returns one row per rank. The next layer sees one hidden row and one full token, so 1 != 1 is false. At n=2, the local length is still one but the comparison is 1 != 2, which is true. At n=3, it is 1 != 3, also true.

This is the article’s unique reproduced result. It proves neither how often a live scheduler creates n=1 on a DP rank nor that every backend corrupts text. It identifies the exact source cell where a shape-only state detector cannot distinguish full from scattered state. That is enough to require a content canary before an eligible graph handles production traffic.

Separate the proven boundary from the issue claim

Section titled “Separate the proven boundary from the issue claim”

vLLM issue #50154 reports garbled GLM-5.2-FP8 completions on TP4, DP2, EP, two B200 nodes, hybrid DP load balancing, and eager execution. The reporter names two nightly builds and shows repeated country names, punctuation, and fragments instead of a clean short completion. That is relevant operational evidence.

The issue’s explanation then says the trigger covers one to three tokens with TP4 because ceil(n/4) == n below TP. The pinned source arithmetic supports that equality only for one token. It does not support the same equality for two or three tokens.

Both statements can coexist if their evidence labels stay visible:

  • Verified here: the old shape guard collides at exactly one full token per DP rank for every audited TP size greater than one.
  • Reported upstream: short prompts produced garbled output in the named TP4/DP2/EP eager environment.
  • Not established here: the per-rank token trace during each bad output, the frequency of failure, or a source proof that every count below TP fails.
  • Required next evidence: record full and scattered token dimensions by DP rank while sweeping one through at least TP active decode tokens.

This distinction matters for incident response. Copying “all batches below TP fail” could cause an operator to chase the wrong threshold. Copying only the n=1 equation could miss a scheduler or backend interaction that the reporter observed but the static audit cannot see. A bounded live sweep resolves both.

The reporter also says eager mode reproduced while one CUDA-graph setting did not, and explicitly leaves the mechanism unknown. Do not treat graph mode as a fix. Test eager and the exact production graph configuration as separate cells, with the same output grader and rank-level occupancy evidence.

Compilation activation is a different experiment again. Before attributing a latency change to compilation, use the GLM-5.2 vLLM torch.compile support audit to pin the installed model class, reject unsupported eager fallback, and require a positive compile receipt plus output parity.

Map stable releases and the nightly window

Section titled “Map stable releases and the nightly window”

Release scope is easy to blur because several related PRs landed within days. The pinned tags separate two different problems:

Runtime boundary Decoder carry guard DP1 activation What the evidence supports
v0.24.0 Later guard absent SP-MoE property requires DP>1 Different implementation; not declared safe or affected
v0.25.0 Old shape guard present Requires DP>1 DP2+EP candidate path exists
v0.25.1 Old shape guard present Requires DP>1 Published before the temporary DP1 removal
Main/nightly after PR #48036 Old shape guard present DP gate removed Brief DP1 memory/performance regression window
Main/nightly after PR #48849 Old shape guard present DP gate restored DP1 regression path closed again
v0.26.0 Old shape guard present Requires DP>1 Published after the restore
v0.27.1 Old shape guard present Requires DP>1 Open fix #50155 not included

PR #48036 removed the data_parallel_size > 1 condition on July 14 while fixing a different DeepSeek-V3.2 MTP accuracy path. Issue #48656 then reported that SP-MoE had silently activated for TP4/EP/DP1. In the reporter’s GLM-5.2-NVFP4 A/B, KV-cache capacity moved from 1,032,448 to 784,300 tokens, while completion throughput moved from 75 to 61 tokens/s at N=1, 407 to 353 at N=8, and 931 to 875 at N=32.

Our arithmetic reproduces the report’s rounded changes as −24.034915%, −18.666667%, −13.267813%, and −6.015038%. Those are reporter measurements on one main/nightly stack, not current v0.27.1 benchmarks and not universal costs of sequence parallelism.

PR #48849 restored the DP gate on July 17. Its separate Nemotron test reported a 5.62 GiB, 7.050558% loading-memory increase in the regression build and a return to its 79.71 GiB baseline after the fix. That is corroborating path evidence, not a GLM-5.2 measurement.

No audited stable tag was published inside the removal-to-restore window: v0.25.1 was released before removal; v0.26.0 was released after restoration. Therefore, do not label all v0.25 or v0.26 stable images with the DP1 regression. Conversely, the restored DP gate does not fix the DP2 low-load shape guard. They are separate incident boundaries.

PR #50155 proposes adding a fallback to the old detector:

Open, unmerged fallback proposed in PR #50155
hidden_states.shape[0] != full_num_tokens
or full_num_tokens < get_tensor_model_parallel_world_size()

That fallback conservatively labels the entire below-TP range as incoming sequence-parallel state. In the source equation, only n=1 changes the boolean because the old shape comparison already returns true for 2 <= n < TP.

At the 2026-08-24 check:

Patch fact Observed state
PR state Open, not draft, unmerged
Last PR update 2026-07-30
Head c163980139bb7d88f03753d40d1a4cce2da1bd62
Code commit 7918bc80bac5d917089022e6da225acf8e5bf0ab
Changed files 1
Diff size 4 additions, 1 deletion
Test files added 0
Fork-head pre-run check Failure
Present in v0.27.1 No

There is another audit discrepancy: issue #50154 describes both the low-token fallback and a residual-shape comparison. The fetched PR file patch contains the guard fallback but not that second proposed change. A PR description is not the deployable diff.

The reporter says the patch restored clean output and approximately 0.94–0.95 GSM8K accuracy on its environment. That result is useful but is not an upstream test, a merged release guarantee, or our measurement. A local fork should be labeled with its own commit, image digest, patch hash, tests, and rollback. “Based on v0.27.1” is not precise enough once source differs.

Pin runtime provenance without importing CUDA

Section titled “Pin runtime provenance without importing CUDA”

Record both package identity and source bytes before launching the model. This Python preflight uses distribution metadata and hashes files without importing vllm, initializing CUDA, or loading weights:

Hash installed vLLM source without importing the runtime
from hashlib import sha256
from importlib.metadata import distribution, version
from pathlib import Path
dist = distribution("vllm")
root = Path(dist.locate_file(""))
print("vllm", version("vllm"))
for relative in (
"vllm/config/parallel.py",
"vllm/model_executor/models/deepseek_v2.py",
"vllm/model_executor/models/utils.py",
):
path = root / relative
print(relative, sha256(path.read_bytes()).hexdigest())

The pinned v0.27.1 source hashes in our receipt are:

File SHA-256
vllm/config/parallel.py 5ee765f980d8e371314caa38a27c5a21ab90f3865e697707d48e608ca4d17f3d
vllm/model_executor/models/deepseek_v2.py bcbd9cde689ffe3f5f271aff3a5e01351b05a68371192cac9d65cb5ffc6b4ba9
vllm/model_executor/models/utils.py f2919647e35f73a8d540e8e5db0fecf2000bb52a6107c39e032a8ddd830e43a1

A wheel may legitimately differ from repository source because of packaging or a downstream patch. A mismatch is a prompt to inspect the installed predicate, not automatic proof of compromise. Archive the container digest, wheel filename and hash, vLLM version string, installed source hashes, checkpoint revision, Transformers version, CUDA/driver stack, graph mode, and every parallel flag in the canary receipt.

Also record resolved topology, not only CLI intent. The official pinned expert-parallel documentation defines EP size from the relevant TP and DP dimensions, and the pinned data-parallel documentation distinguishes internal, external, and hybrid load balancing. The scheduler route determines which DP rank actually reaches one active decode token.

Start from a control that changes one activation condition and nothing else. For example, compare the candidate TP4/DP2/EP/PP1 service with an otherwise identical TP4/DP1/EP/PP1 control. That increases hardware or scheduling cost and does not prove DP1 is generally superior; it isolates the audited DP gate.

Use a fixed prompt corpus with several failure detectors:

  • short facts whose normalized answer is stable;
  • arithmetic with a machine-checkable final value;
  • constrained JSON that must parse and satisfy a schema;
  • a short tool call with an exact tool name and argument shape;
  • multilingual and punctuation-heavy text;
  • a repeated-token trap that exposes loops and phrase collapse; and
  • a longer prompt that verifies the problem is low-load-specific rather than a universal model-quality failure.

Hold these variables fixed in the first comparison: checkpoint bytes, quantization, tokenizer, chat template, reasoning and tool parsers, sampling, maximum output, prefix caching, MTP, all-to-all backend, context cap, and graph mode. Disable optional MTP for the narrow target-path canary, then add it as a separate experiment using the GLM-5.2 MTP guide.

The occupancy sweep must be measured per DP rank. Client concurrency is only an input; hybrid balancing can send a different number of active sequences to each rank. Include at least these phases:

Phase Observed decode rows per DP rank Purpose
Idle-to-one Explicitly capture 1 Exercise the proven guard collision
Below TP Capture 2 through TP−1 where possible Test the reporter’s broader low-batch observation
At TP Capture TP Cross the stated low-batch boundary
Above TP Capture several multiples Establish a healthy-load comparison
Drain Let traffic fall back through one Catch scale-down and queue-drain exposure

Run each phase in eager mode and in the exact production graph mode. Repeat nonstreaming and streaming requests. A graph can mask, reshape, or fail to capture a condition differently; a passing graph cell does not repair a failing eager cell.

Write the manifest before the test:

Minimum SP-MoE canary manifest
{
"checkpoint_revision": "ba978f7d347eaf65d22f1a86833408afdb953541",
"runtime_image_digest": "<record>",
"vllm_source_hashes": "<record>",
"tp_dp_ep_pp": "4_2_true_1",
"all2all_backend": "allgather_reducescatter",
"graph_mode": "eager|production-mode",
"mtp": false,
"per_dp_rank_decode_rows": "<time series>",
"fixture_revision": "<record>",
"control_output_artifact": "<sanitized path>",
"candidate_output_artifact": "<sanitized path>",
"content_parity": "pass|fail",
"protocol_parity": "pass|fail",
"service_slo": "pass|fail",
"rollback": "pass|fail",
"credentials_archived": false
}

The primary gate is answer integrity. Check each candidate response against the control at several levels:

  1. Transport: complete SSE framing, [DONE], finish reason, UTF-8, and no server-side error.
  2. Structure: JSON parses, tool names and arguments validate, reasoning and visible content remain in their expected fields, and streaming deltas assemble into the nonstreaming form.
  3. Deterministic content: exact or normalized equality for fixtures that should be deterministic at temperature zero.
  4. Semantic content: a task-specific grader for answers with harmless wording variation. Preserve the grader version and disagreements.
  5. Degeneration: repeated unigram and n-gram share, punctuation runs, abrupt language switching, duplicated entities, and incomplete fragments.
  6. Distribution: failure count and confidence interval across repeated runs at each observed per-rank occupancy.

Fail closed on any unexplained deterministic mismatch, malformed tool or JSON output, repeated-fragment anomaly, or semantic failure that appears only in the candidate. Do not average a severe corruption into a high overall score. One bad completion in a safety-critical or user-visible path is a debugging event, not “99% accuracy.”

Do not use a second model as the only judge. It may normalize away exactly the repetition or semantic drift being investigated. Machine-check exact fixtures first, then use human review or a versioned independent grader for the remaining cases.

Keep correctness and performance gates separate

Section titled “Keep correctness and performance gates separate”

Sequence parallelism exists to reduce duplicate MoE work and communication. That does not make throughput a proxy for correctness. Apply gates in order:

  1. Provenance gate: exact runtime, patch, model, topology, backend, and modes match the manifest.
  2. Occupancy gate: telemetry proves the canary actually visited one row per DP rank and the broader sweep cells.
  3. Correctness gate: content, structure, and protocol match the control.
  4. Capacity gate: peak HBM, KV slots, graph capture, workspaces, and reserve stay within the declared bound on every rank.
  5. Service gate: TTFT, TPOT, end-to-end latency, tokens/s, queueing, errors, and retries pass separate budgets from idle through load.
  6. Recovery gate: a controlled restart, drain, rank failure, and rollback preserve request and idempotency behavior.
  7. Expansion gate: only a small traffic slice moves first; the observation window covers both peak load and quiet periods.

The historical issue #48656 numbers belong to a brief DP1 activation regression, not this DP2 correctness canary. Do not require current v0.27.1 to reproduce those losses. Instead, compare the exact control and candidate on the workload and source under review.

Quiet-period monitoring is mandatory. A rollout that looks healthy at high concurrency can enter the one-row cell while traffic drains overnight. Segment quality and service metrics by observed rank occupancy so an aggregate daytime graph cannot hide the low-load result.

There is no single universal workaround. Choose one whose cost and semantic change are explicit:

  • Keep DP1 for the affected service replica. This blocks the audited stable activation property but changes replication, capacity, and load balancing.
  • Disable EP for the control. This isolates the EP precondition but can change memory placement and MoE communication substantially.
  • Use another all-to-all backend only as a separate candidate. naive or another non-eligible source path changes expert exchange and performance; it is not a free “disable SP” switch.
  • Carry a local patch. Pin the fork SHA and patch hash, add tests for one through at least TP rows, verify eager and graph modes, and never describe it as released upstream.
  • Wait for an upstream merge and release. Re-audit the final diff, tests, tag, and image; PR #50155’s current patch need not be the code eventually shipped.

Rollback should change one dimension back to the passing control. Pre-pull the control image and keep its model artifact addressable. Test that the router can drain candidate traffic, restore the control, and handle in-flight requests without duplicate side effects. If the canary outcome is ambiguous, mark it failed and preserve artifacts; do not retry with several simultaneous changes.

Switching inference runtimes is not a one-change workaround. If SGLang is a separate candidate, audit its default-on GLM-5.2 DSA prologue with the dedicated SGLang indexer-fusion speed, memory, and rollback guide; do not transfer this vLLM state-guard result to a different execution graph.

For smaller-scale alternatives, the local GLM-5.2 hardware guide separates full checkpoint capacity from practical workstation and rental options. Reducing the experiment’s hardware footprint must not silently change the model or the reader question.

Rent only after the canary contract exists

Section titled “Rent only after the canary contract exists”

A rented cluster is useful when it reproduces the exact GPU count, node split, fabric, image, checkpoint, and DP routing needed for an isolated control. A listing that merely says H200 or B200 is insufficient: TP and EP communication depend on which links stay inside a node, which cross nodes, and whether the chosen all-to-all backend is actually supported.

Cost the failed runs as well as the passing run. A low-load canary may spend more wall time waiting at controlled occupancy than a throughput benchmark. Budget model download, storage, idle GPUs, reruns after an instrumentation failure, and egress. Stop when the manifest cannot be reproduced; expensive ambiguity is not additional evidence.

The candidate and control both produce bad text

Section titled “The candidate and control both produce bad text”

The SP-MoE path is not isolated. Validate the checkpoint and tokenizer hashes, chat template, parser flags, sampling, stop IDs, quantization, and a known-good single-request baseline. A universal failure is not evidence for issue #50154.

Preserve both receipts. This matches the direction of the reporter’s observation but does not establish its mechanism. Do not promote graph mode as a workaround until restart, graph recapture, changing batch shape, and drain cycles keep passing.

That is actionable. Compare its token occupancy, TP rank membership, expert exchange, graph state, device errors, and installed source hashes. Do not hide it in cluster-wide averages or route around it without diagnosing whether the same state can move to another rank.

Two or three rows fail even though the shape equation detects them

Section titled “Two or three rows fail even though the shape equation detects them”

The static collision is not the whole mechanism. Capture actual pre-gather and post-scatter shapes, residual shapes, padding, backend, scheduler state, and graph mode. This would be new evidence supporting the issue’s broader observation rather than a contradiction to suppress.

A single pass is weak evidence. Repeat after warm-up, graph capture, cache growth, traffic drain, restart, and rank redistribution. Confirm telemetry really measured one full decode row on the DP rank; client concurrency one is not the same fact.

Keep it labeled as a fork. Add regression tests, rebase deliberately, and repeat after every upstream change. A passing local canary does not imply PR #50155 will merge unchanged or that a later stable tag contains it.

No. The audited tag retains the old shape-only guard. PR #50155 was open and unmerged at the 2026-08-24 check.

Does every GLM-5.2 vLLM deployment face this path?

Section titled “Does every GLM-5.2 vLLM deployment face this path?”

No. The pinned stable predicate requires an eligible all-to-all backend, EP, TP greater than one, and DP greater than one. The audited decoder-layer path also requires PP1 and an MoE layer. Other configurations have different risks.

Is every decode batch below TP proven to collide?

Section titled “Is every decode batch below TP proven to collide?”

No. For the pinned pad-and-scatter geometry, the old shape guard collides at exactly one full token when TP is greater than one. The upstream reporter makes a broader low-batch observation that should be tested with per-rank traces.

It blocks this exact DP-dependent activation property in audited stable tags. It also changes replica layout, capacity, memory, and scheduling, and it does not certify the rest of the runtime. Treat it as a one-change control or explicit design choice, not a universal patch.

Can I switch CUDA graphs on and declare the issue fixed?

Section titled “Can I switch CUDA graphs on and declare the issue fixed?”

No. The reporter observed a non-reproduction in one graph configuration and said the mechanism was unknown. Test graph and eager modes independently and retain a correctness gate through restarts and changing batch shapes.

Only as a pinned, tested local fork. The fetched patch changes one source file, adds no test file, and does not include every change described in the issue. Add your own regression cells and keep the control image ready.

Alert on any content-parity failure or degeneration anomaly segmented by per-DP-rank decode occupancy, especially a transition to one row. HTTP health, GPU utilization, or aggregate throughput alone can remain normal.

Primary model identity and architecture come from the pinned GLM-5.2-FP8 config, the official model card, and the Z.AI release page. Runtime behavior comes from pinned vLLM v0.24.0–v0.27.1 source files, official EP and DP docs, the GLM-5.2 performance roadmap, and its linked PR chain.

The correctness report is issue #50154; the proposed unmerged change is PR #50155. The separate DP1 activation history comes from PR #47070, PR #47902, PR #48036, issue #48656, and PR #48849. Dates, states, diff sizes, hashes, tag boundaries, and reporter measurements were checked on 2026-08-24.

The deterministic calculator applies the pinned padding and chunk equations to TP2, TP4, and TP8 for token counts 1–16, recomputes reporter percentage deltas, and verifies seven activation cells. Its public JSON receipt contains all twenty-seven source URLs, byte counts, SHA-256 hashes, inputs, rows, boundaries, and invariants. It does not contain credentials or model outputs.

No model, GPU, or upstream test was run for this article. Open-issue output and benchmark values belong to their reporters. Source tests and patches describe code and review scope; they do not prove the deployed service. Recheck upstream state before rollout, because an open PR, tag, source predicate, or official recipe can change after the audit date.