GLM-5.2 vLLM H200 Runtime OOM Workspace Audit

Evan Brooks

AI Infrastructure Editor

Editorial responsibility: GLM52.ai Editorial Team · Published and editorially reviewed August 26, 2026

How we research and review · Report a correction

Disclosed editorial pen name · AI-generated portrait

The operating verdict

Treat vLLM v0.27.1 and audited main 044b0522… as affected for the GLM-5.2-FP8 mixed FlashMLA sparse path on H200. A server can start, report low KV-cache utilization, and still die later when one mixed prefill/decode step requests an unprofiled FP32 accumulator. At 32,768 batched tokens, the pinned one-partition shape is 8 GiB per rank. Temporarily reduce --max-num-batched-tokens, begin with MTP disabled, and require a mixed-traffic burst plus soak test with measured HBM reserve. Treat 8192 as a first canary, not a universal safe default. Both upstream repair paths were open on August 26, so do not turn either PR into an unreviewed production patch.

Original editorial illustration. It separates a growing hidden workspace from a steady KV-cache band; it is not a CUDA trace, HBM capture, vLLM screenshot, or GLM-5.2 benchmark.

A CUDA OOM after sixty-one hours is a different operational problem from a model that fails to load. Startup checks can pass. The KV cache can look mostly empty. Request count can remain modest. Then a single burst combines long prefills with ongoing decodes, fills the scheduler’s token budget, and asks the FP8 sparse-decode kernel for a temporary allocation larger than the physical headroom left on every H200 rank. EngineCore dies and all in-flight work fails.

That sequence is described in vLLM issue #53413 for GLM-5.2-FP8 on 8×H200. It is a contributor report, not our benchmark. The important part is that the reported 8 GiB allocation can be reproduced from the exact source pinned by the latest stable release: vLLM flattens the mixed token set, pads the local query-head envelope to 64, and calls a FlashMLA revision that always allocates FP32 split-KV accumulators.

This guide pins vLLM v0.27.1 at 6e448d0e…, audited main at 044b0522…, FlashMLA at a8f794d1…, and GLM-5.2-FP8 at ba978f7d…. The machine-readable audit receipt hashes 21 public artifacts and regenerates every table below. We made zero model calls, zero local GPU runs, and zero local upstream-test runs.

In this guide

1. Recognize the failure before tuning
2. Pin the affected source path
3. See why TP8 selects mixed FP8
4. Reproduce the hidden workspace math
5. Separate workspace from KV cache
6. Understand the delayed failure
7. Use the token cap as a temporary lever
8. Start the canary without MTP
9. Evaluate prefill-decode disaggregation
10. Treat both upstream fixes as unmerged
11. Build a mixed-traffic canary
12. Measure memory and service together
13. Preserve a one-setting rollback
14. Rent H200 capacity only after preflight
15. Troubleshoot common signatures
16. Frequently asked questions
17. Sources and method

Recognize the failure before tuning

The affected signature is narrow. Do not classify every CUDA OOM as this bug. First match the stack, model, cache path, and traffic shape:

Signal	Affected-workspace evidence	Different problem to investigate
Model	GlmMoeDsaForCausalLM, GLM-5.2-FP8	A dense model, different DSA checkpoint, or different quantization
Backend	FlashMLA sparse FP8 mixed batch	BF16 path, FlashInfer path, SGLang, or another attention backend
Failure site	flash_mla_cuda.sparse_decode_fwd	Weight loading, MoE kernel, graph capture, NCCL, or KV allocation
Timing	Starts successfully; fails on a later mixed burst	Immediate startup failure or deterministic first-request failure
HBM evidence	Requested temporary allocation exceeds physical free HBM	Large reserved-but-unused fragmentation or another process owns memory
KV evidence	KV usage may remain low	A genuine cache-capacity exhaustion tracks admitted context and concurrency
Service result	EngineCore exits; in-flight requests receive EngineDeadError	A recoverable request rejection or scheduler preemption

The issue report names vLLM 0.24.0, PyTorch 2.11, CUDA 13, 8×H200, TP8, expert parallelism, FP8 KV, a 262,144-token model limit, 32 sequences, a 32,768-token batch budget, and five MTP speculative tokens. It reports 8 GiB requested with 6.4 GiB physically free. All eight workers failed on the same step. KV-cache utilization in the preceding minute was reported at only 2.0–5.4%.

Those details are diagnostic anchors, not a transferable performance result. If the stack points at sparse_decode_fwd but the image, source pin, padded head envelope, or scheduler shape differs, recalculate from that exact build.

Pin the affected source path

The first question is not “Which vLLM version is newest?” It is “Which FlashMLA source and mixed-path implementation did this image compile?” A tag can package a vulnerable dependency even if unrelated code changed later.

Our August 26 source matrix is:

Boundary	FlashMLA pin	Full-token mixed path	No-split allocation fix	Decision
vLLM v0.27.1, 6e448d0e…	a8f794d1…	Present	Absent	Treat as affected
Audited main, 044b0522…	a8f794d1…	Present	Absent	Treat as affected
Open vLLM PR #49357	Existing dependency plus vLLM chunking	Bounded by chunks	Alternative workaround	Source evaluation only
Open FlashMLA PR #19 + vLLM PR #53755	Proposed new dependency	Full call retained	Proposed conditional allocation	Wait for merge and packaging

The v0.27.1 flashmla.cmake and the audited-main flashmla.cmake name the same a8f794d1… revision. The bundled sparse_decode.h allocates lse_accum and o_accum before every sparse-decode kernel run, then runs the combine kernel. There is no one-partition exception in that source.

This is why “try current main” is not a fix. The current recommendation is not “use a newer nightly.” Resolve and archive the actual dependency commit before spending H200 time.

See why TP8 selects mixed FP8

The pinned GLM-5.2-FP8 config declares 64 global query heads. TP8 leaves eight query heads per rank. In v0.27.1, MIN_HEADS_FOR_BF16_PREFILL is 32. The FlashMLA sparse builder uses the mixed FP8 path whenever the per-rank count is below that threshold.

The source explains the choice: the separate path upconverts the FP8 cache for a BF16 prefill kernel whose head padding is expensive at low per-rank head counts. The mixed path instead treats all prefill and decode tokens as one batch and sends them through the FP8 decode kernel.

That path then pads the eight actual heads to a supported kernel envelope of 64. Its call uses a 512-wide internal value dimension. The chain is therefore:

Affected GLM-5.2 TP8 source path

64 global query heads / TP8 = 8 local heads
8 local heads < mixed-path threshold 32
mixed FP8 path flattens all scheduled tokens
kernel pads 8 local heads to 64
FlashMLA allocates FP32 split-KV accumulators for the padded shape

Expert parallelism does not remove this attention allocation. EP changes MoE expert placement and communication; the attention backend still receives its rank-local query and mixed token batch. Our GLM-5.2 expert-parallelism guide covers that separate topology and load-balance decision.

Reproduce the hidden workspace math

For the issue’s one-batch, one-SM-partition shape, the dominant accumulator is deterministic from the bundled FlashMLA allocation:

Dominant no-split FP32 accumulator

bytes per scheduled token
= (batch + SM partitions)
 × padded query heads
 × kernel value dimension
 × FP32 bytes

= (1 + 1) × 64 × 512 × 4
= 262,144 bytes
= 0.25 MiB per scheduled token

The resulting table is unique to this pinned source shape:

max-num-batched-tokens	Dominant FP32 accumulator	LSE accumulator	BF16 output shape	Accumulator + output
4,096	1 GiB	2 MiB	0.25 GiB	1.25 GiB
8,192	2 GiB	4 MiB	0.5 GiB	2.5 GiB
16,384	4 GiB	8 MiB	1 GiB	5 GiB
32,768	8 GiB	16 MiB	2 GiB	10 GiB
65,536	16 GiB	32 MiB	4 GiB	20 GiB

The 32,768 row reproduces the reported 8 GiB attempted allocation. The issue also gives an equivalent model-geometry observation: 32768 × 2048 index top K × 128 index dimension × 1 FP8 byte = 8 GiB. The equality helps reconcile the report, but the table above follows the actual FP32 accumulator shape in FlashMLA rather than asserting that every index tensor becomes that workspace.

Do not use the last column as a full HBM forecast. It excludes weights, scale metadata, KV and index caches, graph pools, MoE and communication workspaces, allocator reserve, scheduler metadata, outputs held elsewhere, fragmentation, NCCL, CUDA context, and other processes. Its purpose is to reject a token cap that cannot fit even the named dominant allocation.

The public receipt regenerates these numbers with:

Verify the checked-in arithmetic

node docs/evidence/glm-5-2-vllm-h200-runtime-oom-2026-08-26/calculate.mjs \
  > /tmp/glm-5-2-vllm-h200-runtime-oom-results.json

diff -u \
  docs/evidence/glm-5-2-vllm-h200-runtime-oom-2026-08-26/results.json \
  /tmp/glm-5-2-vllm-h200-runtime-oom-results.json

Separate workspace from KV cache

--gpu-memory-utilization and KV-cache utilization answer different questions. In vLLM v0.27.1, the GPU utilization field sets a per-instance model-executor fraction, and vLLM can infer KV-cache bytes from the remaining profiled budget. A live KV percentage tells you how much of that cache pool is occupied. Neither metric guarantees that an unprofiled transient kernel workspace fits in physical HBM at the largest scheduler step.

Low KV occupancy does not prove dynamic workspace safety. Read it beside physical free HBM, allocator peaks, graph pools, and scheduler tokens.

The reported failure makes the distinction concrete:

Observation before failure	What it proves	What it does not prove
KV cache at 2.0–5.4%	Most allocated KV blocks were unused at that moment	Eight GiB of contiguous physical headroom exists
At most eight running requests	Sequence count was modest	Scheduled tokens were modest during a mixed prefill burst
Server had run about 61 hours	Startup and many earlier batches worked	The worst allowed batch shape was exercised
6.4 GiB physically free	Some headroom remained	The pending 8 GiB accumulator plus other allocations fits
5.01 GiB reserved but unallocated	PyTorch had reserved segments	Those segments could satisfy the exact request without fragmentation or pool constraints

The issue’s 8 GiB request minus 6.4 GiB free leaves a 1.6 GiB physical deficit before any safety reserve. Raising the configured utilization would move in the wrong direction because it lets the executor consume more HBM. Lowering it may create headroom, but the direct source-shape lever is the scheduler token cap. Measure both rather than treating either as a guarantee.

This is also distinct from our vLLM decode context parallelism gate. DCP changes sequence-cache replication and distributed attention. It does not make an unbounded no-split mixed workspace disappear, and a cache-capacity gain can leave less physical reserve if it is immediately filled.

Understand the delayed failure

A limit only matters when traffic reaches it. The affected server can serve small decodes and short prefills for days because its actual token set stays well below 32,768. The failure becomes plausible when a long chunked prefill arrives while existing sequences are decoding.

With five MTP speculative tokens, each active decode sequence can contribute one target token plus five drafts. At the configured 32-sequence ceiling, a decode-only step contributes roughly:

Decode-only MTP occupancy

(1 target + 5 speculative tokens) × 32 sequences = 192 tokens

That is far below 32,768. The remaining 32,576-token envelope can be filled by mixed prefills. This explains why a benchmark that sends only short prompts or steady decodes may never touch the dangerous shape. It also explains why max_num_seqs=32 is not a memory bound for this workspace.

The exact scheduler can append or reserve tokens differently across versions. The v0.27.1 scheduler source explicitly distinguishes maximum batched tokens from maximum scheduled tokens for models that append speculative tokens. Treat the 192 calculation as a workload-design check, not a promise about every internal batch.

Use the token cap as a temporary lever

Reducing --max-num-batched-tokens lowers the dominant source shape linearly: 32K gives 8 GiB, 16K gives 4 GiB, and 8K gives 2 GiB. An operator commenting on issue #53413 reports using 8,192 on a four-H200 path for roughly two months without this OOM. That is useful corroboration, but it is not an upstream guarantee, a controlled comparison, or evidence for your image and workload.

Treat 8192 as a first canary, never as a copied production default.

Use 8,192 as a first canary only if it satisfies three conditions:

1. The exact build still selects the affected path, so the test is relevant.
2. Peak physical free HBM remains above the calculated workspace plus an explicit reserve during repeated mixed bursts.
3. Lowering the cap does not cause an unacceptable TTFT, throughput, fairness, or queueing regression for the real workload.

A bounded starting command changes one lever from the reported profile:

Illustrative temporary canary delta

vllm serve /models/GLM-5.2-FP8 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192

This is not a complete deployment command. Pin the image digest, checkpoint revision, driver, CUDA, PyTorch, parsers, networking, security, storage, and all other production arguments separately. MTP is intentionally absent from the first control.

Do not switch to BF16 KV merely because the reported FP8 path fails. Issue #44545 reports gibberish in its BF16 comparison on an adjacent GLM checkpoint. A cache-dtype change affects capacity and numerical behavior and needs its own output-parity corpus. One test should change one variable.

Start the canary without MTP

MTP is not identified as the root cause. It can make decode steps wider and changes the token scheduler, CUDA graph shapes, acceptance behavior, latency, and output-validation surface. Removing it first creates a simpler control.

Follow a two-phase design:

Phase	MTP	Token cap	Required result
Control	Disabled	One bounded value, such as 8,192	Stable mixed burst, soak, output corpus, restart, and HBM reserve
Candidate	Five speculative tokens	Same cap	Same correctness and recovery, useful accepted-token speed, no reserve regression

Measure accepted speculative tokens, not draft volume. A low acceptance rate can add work and graph pressure without useful decode speed. Our GLM-5.2 MTP guide explains the acceptance and rollback metrics. Do not combine MTP enablement with a token-cap increase; if the candidate fails, the cause must remain attributable.

Evaluate prefill-decode disaggregation

A vLLM maintainer names prefill-decode disaggregation as another workaround if the GPU budget allows. The architecture separates long prefill batches from decode workers, so the decode side does not receive the same mixed token shape. That can avoid this exact trigger, but it is not free capacity.

The current vLLM Ascend GLM-5.2 guide documents a decode-side relationship of (MTP speculative count + 1) × maximum sequences for one high-throughput profile. With five drafts and 32 sequences, that is 192 batched tokens on the decode side. This is official evidence for that Ascend recipe, not an H200 configuration to copy.

The llm-d GLM-5.2 H200 guide also composes a prefill/decode-disaggregated H200 route. It establishes that a maintained ecosystem path exists. It does not prove your KV-transfer backend, fabric, routing, failure recovery, or cost.

Evaluate disaggregation only after recording:

• prefill and decode GPU counts, topology, HBM, and image digests;
• KV transfer backend, devices, bandwidth, timeouts, and ownership;
• admission policy and token limits on both sides;
• TTFT, TPOT, accepted throughput, queue depth, transfer latency, and failures;
• behavior when a prefill or decode rank dies mid-transfer;
• restart ordering, orphan cleanup, retry semantics, and rollback to the single-engine topology; and
• the incremental cost versus keeping a lower token cap.

If one node and a bounded cap meet the service objective, disaggregation may add more risk than it removes.

Treat both upstream fixes as unmerged

There are two repair designs, and they should not be conflated.

vLLM PR 49357 chunks the mixed batch

PR #49357 proposes a VLLM_FLASHMLA_SPARSE_MAX_SCRATCH_MB bound, splits a large mixed token set, and runs the kernel over multiple chunks. Its author calls the approach a “bandaid” and acknowledges a throughput tradeoff. At the audit time the PR was open and had merge conflicts reported by automation.

Chunking is attractive because it limits the allocation even when a split workspace remains necessary. It also changes launch count and aggregation behavior. Correctness, peak allocation, throughput, TTFT, cancellation, and graph compatibility all require testing on the rebased final code.

FlashMLA PR 19 removes an unused no-split workspace

FlashMLA PR #19 checks whether more than one SM partition exists. In the one-partition case it avoids allocating the FP32 accumulators and returns the kernel output directly. vLLM PR #53755 changes the bundled dependency to integrate that work.

The integration PR reports one H100 80GB test where its small no-split case fell from 42.3467 MiB peak allocation to 8.2822 MiB, equal to output tensors within the stated one-MiB bound. It also reports ten passing vLLM kernel tests and an FP8 sparse-decode equivalence test. Those are PR-author results on a small H100 fixture, not an 8×H200 GLM-5.2 soak.

An open PR is not a release. As of the audit, the PR titles and bodies also retain explicit review cautions. Wait until the chosen repair is merged, bundled by a pinned vLLM commit or immutable image, and independently validated on the intended model and workload. Do not cherry-pick both approaches at once and call the result upstream vLLM.

Do not deploy either open PR as a production fix.

Build a mixed-traffic canary

A useful canary must reach the dangerous scheduler shape repeatedly. A single health prompt, vllm bench decode-only run, or startup memory report cannot do that.

Freeze this contract before allocation:

Dimension	Required pin or workload
Software	Image digest; vLLM, FlashMLA, PyTorch, CUDA, driver; checkpoint revision
Topology	Eight H200 GPUs; TP8; EP setting; interconnect; one process map
API behavior	Reasoning and tool parsers; streaming; stop conditions; cancellation
Control	MTP off; fixed token cap; fixed model length and sequence cap
Prefill load	Short, medium, and long prompts arriving while decodes are active
Decode load	Fixed output lengths, early stops, tool turns, and cancellations
Burst	Enough simultaneous arrivals to hit the scheduler cap many times
Soak	Longer than normal traffic cycles and scheduled maintenance boundaries
Failure	Cancel, client disconnect, worker failure, rank restart, and full restart
Rollback	Restore previous cap and MTP state in the identical image

Use a fixed output corpus for text, reasoning, tools, long context, and stop tokens. A memory fix that changes outputs is not a successful memory fix. Archive only sanitized prompts or hashes when production data is sensitive.

For the old and candidate configurations, run the same arrival trace several times. Random load generators can hide rare batch shapes. Preserve the scheduler token count and request composition at every measured peak so the largest allocation can be explained rather than merely graphed.

Measure memory and service together

Capture whole-GPU and allocator views. Each answers a different question:

Metric	Why it matters	Rejection signal
Physical free HBM per rank	Can the next non-PyTorch or PyTorch allocation fit?	Minimum reserve falls below the declared workspace plus safety margin
PyTorch allocated and reserved	Separates live tensors from cached segments	Persistent growth, large unexplained pools, or fragmentation pattern
CUDA graph private pools	Graph shapes can hold memory outside ordinary peaks	New cap/MTP arm expands pools beyond reserve
KV and index cache bytes and occupancy	Prevents mislabeling cache pressure as workspace pressure	Cache fills or capacity falls below workload need
Scheduler batched and scheduled tokens	Links the memory peak to the controlling input	Peak cannot be reproduced or explained
TTFT and TPOT distributions	Lower caps may trade throughput for latency or queueing	Tail objective fails even though OOM disappears
Accepted output throughput	Rejects faster invalid or speculative work	Accepted-token gain is absent or output corpus fails
Engine errors and recovery time	An OOM is a service incident, not just a memory event	EngineDeadError, stuck requests, or ambiguous restart state

Do not average away the failure. The issue occurred after a long healthy period, so minimum free HBM, maximum workspace, p99/p99.9 latency, error counts, and worst recovery matter more than a mean dashboard tile.

Promote only when every rank maintains the declared reserve at the worst repeated mixed shape and service-level metrics remain useful. H200 capacity is expensive; a canary that cannot explain its worst batch is not ready to run unattended.

Preserve a one-setting rollback

The temporary control should be reversible without changing image, checkpoint, cache dtype, parsers, topology, or request corpus. Record the original and candidate token caps and whether MTP is enabled. Exercise a restart in both directions before promotion.

Use this rollback sequence:

1. Stop admission and record the exact request and scheduler state.
2. Drain or fail in-flight work according to an explicit client contract.
3. Stop only the intended deployment and confirm all ranks exit.
4. Restore the last accepted token cap and MTP state in the same pinned image.
5. Start all ranks and verify backend, checkpoint, parsers, cache dtype, and topology from logs and a harmless output corpus.
6. Reopen admission gradually and confirm free-HBM reserve and tail latency.
7. Preserve the failed candidate receipt; do not overwrite it with the successful rollback run.

If rollback requires an unreviewed dependency patch, cache conversion, or a different topology, it is not a one-setting rollback. Keep the safer prior configuration until the upstream fix is packaged.

Rent H200 capacity only after preflight

A rented H200 node is useful only after the source and test contract are frozen. Renting does not make an open PR merged, guarantee eight GPUs share the required fabric, or prove that a marketplace image contains the pinned vLLM and FlashMLA revisions.

Before opening a rental, record the GPU count and HBM, NVLink/NVSwitch topology, driver/CUDA, base image digest, checkpoint source and storage size, network and egress needs, maximum hourly spend, evidence path, teardown plan, and a hard stop if the requested topology differs. Do not put production secrets or private prompts into a marketplace instance merely to reproduce a public kernel bug.

Run a bounded GLM-5.2 H200 workspace canary

Rent only after the affected source path, one-variable canary, memory reserve, output corpus, spend ceiling, and rollback are fixed in writing. Begin with MTP off and a bounded token cap, run repeated mixed prefill/decode bursts, archive sanitized receipts, and delete only the task-owned resources after the rollback exercise.

Check current H200 availability on RunPod →

Affiliate disclosure: This is an affiliate link and a referral link. GLM52.ai may receive a commission or referral credit after qualifying usage at no added charge from us. A rental does not verify that FlashMLA PR #19, vLLM PR #49357, PR #53755, or any successor has merged; that an image contains a fixed dependency; or that GPU count, HBM, fabric, driver, CUDA, vLLM, FlashMLA, checkpoint, price, storage, egress, security, and deletion match this audit. Verify every live term and pin before use.

If the eight-H200 test is not justified, keep the lower cap, use a documented hosted route, or revisit the broader GLM-5.2 local deployment decision. Hardware spend should answer a frozen decision, not substitute for source review.

Troubleshoot common signatures

The server OOMs during model loading

That is not the delayed mixed workspace signature. Check checkpoint precision, tensor and expert placement, quantization scales, loading duplication, graph capture, and processes already holding HBM. Do not lower the scheduler token cap and declare a load-time problem fixed.

The stack ends in sparse_decode_fwd on a long first prompt

Match the FlashMLA pin, cache dtype, mixed-path selection, padded heads, token count, and physical free HBM. The adjacent GLM-5.1 report in issue #44545 reproduced the same 8 GiB signature with a long input. A first-request failure and a 61-hour failure can share the same allocation if their first worst-case batch occurs at different times.

KV cache is almost empty, so monitoring says memory is healthy

Fix the dashboard. Show physical free HBM, allocated and reserved bytes, graph pools, scheduler tokens, and backend workspace peaks beside KV occupancy. A cache-only panel cannot diagnose this failure.

Lowering the token cap prevents OOM but TTFT regresses

That is an expected tradeoff candidate, not proof the cap is wrong. Compare several bounded values under identical arrivals, report p50 through p99.9 TTFT and TPOT, and choose the smallest cap that satisfies both reserve and service objectives. If none does, evaluate disaggregation or a different serving route.

A nightly image starts and survives the smoke test

Resolve its vLLM and FlashMLA commits. Confirm whether the repair merged or the image carries a private patch. Then rerun mixed bursts, soak, failure, output, and rollback gates. “Nightly” is a moving label, not provenance.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True removes another OOM

That setting can help fragmentation patterns, but it cannot make an 8 GiB allocation fit into 6.4 GiB of physical free HBM. Preserve it only if an A/B shows the actual failure was allocator fragmentation and the source-shape workspace plus reserve already fit.

Frequently asked questions

Is vLLM v0.27.1 safe for GLM-5.2 FP8 on 8xH200?

Not for this path by version number alone. At the August 26 audit, v0.27.1 bundled the affected FlashMLA commit and retained the mixed full-token path. Use a bounded token cap and production-shaped canary, or wait for a merged and pinned fix. Other GLM-5.2 topologies may select different paths and require their own audit.

Why can the OOM happen when KV-cache utilization is low?

Because the failed allocation is a transient FP32 sparse-decode accumulator, not a request for more KV blocks. KV occupancy describes one reserved pool. The kernel still needs physical HBM for workspaces, outputs, graph pools, and other state.

Is --max-num-batched-tokens 8192 guaranteed safe?

No. It reduces the audited dominant accumulator shape from 8 GiB to 2 GiB, and one issue commenter reports operating at that cap. Actual safety depends on the resolved source path, scheduler, other allocations, physical reserve, traffic, graph shapes, and concurrent processes. Treat it as a first canary.

Should I reduce --gpu-memory-utilization instead?

It may preserve more physical reserve, but it is an indirect lever and can reduce KV capacity. The token cap directly bounds the audited allocation shape. Test a one-variable cap first, then adjust the overall memory fraction only with explicit cache-capacity and service measurements.

Does disabling MTP fix the bug?

No. The unbounded workspace is the root source condition. Disabling MTP makes the control simpler and reduces decode token expansion. Add MTP back only after the plain mixed-batch arm passes memory, output, service, soak, and rollback gates.

Should I cherry-pick FlashMLA PR 19 or vLLM PR 49357?

Not into production based on this guide. Both were open. One removes an unused one-partition workspace; the other chunks mixed batches with a potential throughput cost. Wait for review and merge, pin the packaged source, then test the final code on GLM-5.2 and the intended H200 workload.

Is prefill-decode disaggregation always better?

No. It can keep long prefills off decode ranks and is an upstream-suggested workaround, but it consumes more infrastructure and adds KV transfer, routing, capacity, and recovery failures. Choose it only when those costs beat a bounded single-engine cap.

Sources and method

This is a pinned-source and deterministic arithmetic audit, not a live GPU benchmark. The primary sources are:

• Z.ai’s GLM-5.2 release and the Zhipu research index — mandatory first-party discovery checks;
• the pinned GLM-5.2-FP8 model card and config;
• vLLM v0.27.1, its pinned FlashMLA sparse backend, scheduler config, cache config, and GLM-5.2 recipe;
• issue #53413, the adjacent issue #44545, open vLLM PR #49357, open FlashMLA PR #19, and open vLLM PR #53755;
• the current vLLM Ascend GLM-5.2 guide and llm-d H200 guide for separately bounded disaggregation evidence.

AI HOT was used only as untrusted discovery intake. Its batch included LangChain/Airbyte, OpenWorker security agents, and an OpenRouter model selector. Those leads were rejected for weak GLM-5.2 linkage or overlap with existing retrieval, agent, and routing pages. The final topic came from proactive inspection of the official vLLM and FlashMLA repositories.

The research record is summarized here; the checked-in file preserves intake classification, overlap audit, source boundaries, formulas, repair status, canary gates, and limitations. The public JSON receipt contains 21 source URLs, byte counts, SHA-256 hashes, five workspace rows, the reported 1.6 GiB physical deficit, version boundaries, and invariants. Public search was used only to inspect result supply. No paid SerpAPI request was made because the site’s monthly ceiling was already exceeded.

We did not call GLM-5.2, rent an H200, execute upstream tests, reproduce the 61-hour workload, or measure an allocation. Issue and PR observations remain attributed to their authors. No model output, API key, cookie, credential, private prompt, or private infrastructure detail is stored. Recheck PR and release status before acting because the repair boundary can change after the publication date.

About the editorial role

Evan Brooks

AI Infrastructure Editor

Evan Brooks serves as GLM52.ai’s disclosed editorial pen name for infrastructure, deployment, and training guides. The role examines checkpoint size, memory math, provider limits, implementation evidence, and operational risk. The GLM52.ai Editorial Team produces and fact-checks every article under this byline.

The GLM52.ai Editorial Team accepts responsibility for this article. Read our editorial policy or request a correction.

An AI image tool created this portrait. It does not depict a real individual.