GLM-5.2 DSA Cache LayerSplit: SGLang Deployment Audit

Evan Brooks

AI Infrastructure Editor

Editorial responsibility: GLM52.ai Editorial Team · Published and editorially reviewed August 23, 2026

How we research and review · Report a correction

Disclosed editorial pen name · AI-generated portrait

The deployment verdict

Treat SGLang DSA cache LayerSplit as a bounded PD-prefill pilot for GLM-5.2, not a general memory switch. Pin SGLang v0.5.18, use a PD-disaggregated prefill role, enable prefill context parallelism with the interleave strategy, set attention CP above one, keep pipeline parallelism at one, and use Mooncake or its TCP transport. Leave the flag off on unified and decode workers. The decode side retains an ordinary full cache and retrieves fragments from every prefill CP rank. Promote only after live owner-broadcast, all-rank transfer, output-parity, memory, service-metric, and rollback evidence.

Original editorial illustration: four prefill ranks own contiguous cache-layer ranges, a single remote scratch cell receives the active owner broadcast, and the transfer bridge leads to a separate full-cache decode pool. Amber branches represent configurations that should fail preflight. It is not a benchmark.

SGLang’s DSA cache LayerSplit changes where the prefill worker stores request-dependent DeepSeek Sparse Attention cache state. Without the feature, each context-parallel rank can materialize the cache for all transformer layers. With LayerSplit, a rank materializes only its assigned contiguous layers plus one full-layer remote scratch buffer. When execution reaches a layer owned by another rank, the owner broadcasts that layer into the scratch buffer so every rank can continue the attention step.

That sounds like a local allocation option. It is not. The feature sits at the intersection of GLM-5.2’s DSA architecture, prefill context parallelism, PD disaggregation, inter-rank broadcast, and the prefill-to-decode transfer protocol. A command can start and still miss the intended saving at CP1. A decode command can be invalid even when its prefill peer is valid. A measured cache-cell reduction can be accurate while total HBM barely moves. And an upstream test file can exist without proving that a different image, network, or topology passes.

This audit pins SGLang v0.5.18 at commit 71de97b264b04dcd514cf904003028aefe9775c8 and the official GLM-5.2-FP8 revision ba978f7d347eaf65d22f1a86833408afdb953541. Its machine-readable receipt hashes nineteen public sources and reproduces three cache-layer plans. We made zero model calls and zero GPU runs. Registered upstream tests are described as coverage, not reported as executions.

In this guide

1. Make the topology decision first
2. Understand what LayerSplit changes
3. Apply every eligibility constraint
4. Reproduce the 78-layer shard plan
5. Bound the memory claim correctly
6. Keep prefill and decode asymmetric
7. Preflight a pinned launch
8. Read registered tests without overclaiming
9. Build a live canary matrix
10. Measure correctness, memory, and service impact
11. Inject failures and preserve rollback
12. Rent only after fixing the experiment
13. Troubleshoot common failures
14. Frequently asked questions
15. Sources and method

Make the topology decision first

LayerSplit is worth evaluating only when a PD prefill pool is constrained by the DSA KV/indexer cache cell and already benefits from context parallelism. It does not reduce checkpoint weight memory. It does not shard the decode cache. It does not turn a unified server into a PD deployment. It also adds an owner-broadcast path whose cost and correctness must be observed under the same long-context distribution the service will receive.

Use this decision table before copying a flag:

Proposed deployment	Source-level outcome at the pinned revision	Decision
GLM-5.2 DSA, PD prefill, CP4, interleave, Mooncake, PP1	Meets the narrow structural contract	Eligible for an isolated pilot
Unified worker that performs both prefill and decode	LayerSplit role is invalid	Reject before launch
PD decode worker with the LayerSplit flag	Decode keeps ordinary cache semantics	Remove the flag from decode
PD prefill with zigzag CP	Strategy is outside the feature contract	Reject or change to interleave
PD prefill with NIXL or Mori	Transfer backend is unsupported for this path	Reject; do not infer parity from other PD features
PD prefill with PP2	CP plus PP is not validated here	Reject the first pilot
DSA prefill with CP1	Starts with ordinary non-sharded semantics	No LayerSplit saving; use as a negative control
A model without the DSA architecture	The feature predicate is false	Reject; this guide does not apply

The positive row is deliberately narrow. “SGLang supports GLM-5.2” is not enough, because support for the model does not imply support for every parallel layout. “Mooncake supports PD transfer” is also not enough, because a working transfer backend does not prove cache-layer ownership or the all-rank fragment contract. Each dimension must match in one resolved launch.

A practical go/no-go sequence is:

1. Profile LayerSplit off and verify that the relevant prefill cache cell, not weights or another allocation, is the limiting memory.
2. Freeze the SGLang commit, model revision, container digest, driver, GPU topology, PD layout, request corpus, and metric definitions.
3. Run startup-negative controls for role, CP strategy, backend, PP size, and CP1 semantics before allocating a production-shaped cluster.
4. Establish cache and answer parity with the flag off versus on.
5. Measure total HBM, the named cache pool, TTFT, prefill throughput, transfer time, failures, and tails under repeated trials.
6. Rehearse one-flag rollback with the same router and decode pool.

If the cache cell is not the bottleneck, stop. A feature can work exactly as designed and still be the wrong optimization.

Understand what LayerSplit changes

The pinned dsa_cache_layer_split.py describes an ownership pool layered around the core KV-cache types. The base pool remains intact; LayerSplit adds the mapping from transformer layer to owner rank, the local-layer materialization, a remote scratch slot, and the broadcast operation.

For 78 decoder layers and CP4, ownership is contiguous:

CP rank	Owned half-open range	Human-readable layers	Owned count
0	[0, 20)	1–20	20
1	[20, 40)	21–40	20
2	[40, 59)	41–59	19
3	[59, 78)	60–78	19

Suppose the forward pass needs zero-indexed layer 47. Rank 2 owns that layer, so its local buffer is the source. The CP communicator broadcasts the relevant cache data. Ranks 0, 1, and 3 receive it in their remote scratch slot. When the next remote layer is needed, the same slot can be reused; a rank does not keep an extra copy of every unowned layer.

This design explains both the memory opportunity and the new risk. The allocation removes most full-layer cells from each rank, but correctness now depends on identifying the same owner, selecting the right layer view, executing communication on the intended stream, and consuming the completed broadcast before the scratch buffer is reused.

The current source explicitly creates a PyNccl communicator for this path. That detail matters because a missing communicator could turn what looks like a broadcast call into a no-op or leave a stale scratch buffer. A production canary should still prove the rendered behavior from logs and output parity; the presence of the communicator in source is necessary evidence, not an end-to-end result.

LayerSplit affects request cache state, not parameters. GLM-5.2 weights remain replicated or sharded according to the ordinary TP/EP/PP model layout. CUDA graphs, allocator reserve, attention workspaces, expert buffers, request tables, and transfer staging are also outside this layer-equivalent calculation. Do not label the feature a “75% HBM reduction.”

Apply every eligibility constraint

The pinned server_args.py exposes the LayerSplit flag and bounds its transfer backend. The pinned cp/utils.py then resolves whether a model runner actually uses the sharded pool.

Check the constraints as a conjunction:

Constraint	How to verify it	Failure meaning
DSA architecture	Pinned model config says GlmMoeDsaForCausalLM	A non-DSA model cannot enter the path
PD prefill role	Resolved disaggregation mode is prefill	Unified or decode role is invalid
Prefill CP enabled	--enable-prefill-cp is present	No prefill CP group exists for the feature
Interleave strategy	--cp-strategy interleave	Zigzag is not an accepted substitute
Attention CP greater than one	Resolve the actual attention CP size	CP1 falls back to ordinary full-layer sizing
Mooncake transfer	Backend resolves to mooncake; TCP is normalized	NIXL and Mori are not covered
PP size one	Resolve pipeline parallel size	CP+PP is outside the validated boundary
Non-draft worker	Model runner is not the MTP draft worker	Draft cache remains full by design

The official pinned GLM-5.2-FP8 config provides the model-side facts: 78 decoder layers, one next-token/MTP layer, a 1,048,576-position maximum, and DSA fields including an index top-k of 2,048, index head dimension 128, KV LoRA rank 512, and RoPE head dimension 64. Those facts establish DSA eligibility and the layer count. They do not establish a safe token limit, cache size, or throughput for a particular machine.

Mooncake TCP is a transport selection, not another cache algorithm. At this revision, mooncake_tcp is normalized to the Mooncake backend while forcing TCP and skipping RDMA device selection. Record the resolved backend, not just the text typed on the command line. If the experiment needs NIXL or Mori for operational reasons, LayerSplit is not the right pilot until the exact pinned source supports and tests that combination.

Finally, MTP needs precise language. The registered end-to-end scenario includes GLM-5.2 PD plus MTP, but the draft worker is explicitly excluded from LayerSplit and keeps a full cache. “LayerSplit supports MTP” must not be translated into “every MTP cache is sharded.”

Reproduce the 78-layer shard plan

The partition helper divides the total as evenly as possible into contiguous ranges. It also sizes the per-rank pool to the ceiling of layers divided by CP plus one remote scratch layer. The following dependency-free script reproduces the plan without importing SGLang:

plan-glm52-layer-split.mjs

const layers = Number(process.argv[2] ?? 78);
const cp = Number(process.argv[3] ?? 4);

if (!Number.isInteger(layers) || layers < 1) throw new Error('layers must be positive');
if (!Number.isInteger(cp) || cp < 2) throw new Error('CP must be at least two');

const base = Math.floor(layers / cp);
const extra = layers % cp;
let start = 0;
const ranges = [];

for (let rank = 0; rank < cp; rank += 1) {
  const owned = base + (rank < extra ? 1 : 0);
  const end = start + owned;
  ranges.push({ rank, start, end, owned });
  start = end;
}

if (start !== layers) throw new Error('incomplete layer coverage');
const covered = ranges.flatMap((r) =>
  Array.from({ length: r.owned }, (_, offset) => r.start + offset)
);
if (new Set(covered).size !== layers) throw new Error('duplicate or missing layer');

const maxOwned = Math.max(...ranges.map((r) => r.owned));
const layerEquivalents = maxOwned + 1;
const reduction = 100 * (1 - layerEquivalents / layers);

console.table(ranges);
console.log({ layerEquivalents, scratchIncludedUpperBoundReduction: reduction });

Run the three candidate plans:

Reproduce CP2, CP4, and CP8 ownership

node plan-glm52-layer-split.mjs 78 2
node plan-glm52-layer-split.mjs 78 4
node plan-glm52-layer-split.mjs 78 8

The reviewed outputs are:

CP size	Owned counts	Maximum owned layers	Plus scratch	Scratch-included theoretical upper-bound reduction
2	39, 39	39	40	48.7179%
4	20, 20, 19, 19	20	21	73.0769%
8	10, 10, 10, 10, 10, 10, 9, 9	10	11	85.8974%

“Upper bound” is essential. The calculation assumes equal per-layer cache-cell geometry and compares the full 78-layer cell with the maximum owned range plus one scratch layer. Runtime alignment, metadata, staging, allocator granularity, and non-cache allocations can reduce the observable saving. CP8 also adds more communication participants; a larger theoretical cache reduction is not automatically a better latency or throughput result.

Validate the actual startup log against the planned half-open ranges. A balanced count alone is insufficient: all 78 layers must appear exactly once, no two ranks may claim the same layer, and every rank must agree on CP group membership. Archive the rank, range, model revision, SGLang commit, and resolved arguments with the trial.

Bound the memory claim correctly

SGLang’s v0.5.16 release introduced the feature with a concrete GLM-5.2-FP8 cell: 8,192 tokens, CP4, 0.77 GB per rank before LayerSplit, and 0.20 GB per rank after. The calculated decrease is:

Reproduce the named publisher cell

(0.77 - 0.20) / 0.77 × 100 = 74.026%
four-rank aggregate: 3.08 GB → 0.80 GB

That result is consistent with the roughly 74% description and close to the 73.0769% scratch-included layer-equivalent bound. It is still only one named cache cell. The source does not make that number a whole-server memory result, a throughput result, or a guarantee for a different token count, cache dtype, CP degree, allocator state, or SGLang revision.

Report at least four separate memory lines:

Metric	Why it is needed	Acceptable label
DSA KV/indexer cache pool requested bytes	Closest measure of the feature’s target	LayerSplit cache-pool delta
Per-rank allocated and reserved device memory	Shows allocator response and headroom	Device-memory delta
Whole-process and whole-GPU peak HBM	Catches workspaces and concurrent processes	Total-HBM observation
Maximum admitted tokens or requests at fixed reserve	Tests whether saved capacity becomes useful	Capacity result under fixed config

Never subtract two dashboard screenshots without fixing the workload phase. Capture after model load, after graph capture, after cache-pool allocation, during the same prefill corpus, and at steady state. Record all ranks; a mean can hide one rank whose extra staging or imbalance removes the headroom.

The official SGLang GLM-5.2 cookbook describes LayerSplit as able to reduce KV cache memory by up to 75% on the prefill side. “Up to” and “prefill side” should remain attached whenever the claim is summarized. Our deterministic CP8 percentage is not a contradiction; it is a source-formula upper bound for layer equivalents, not another measured publisher cell.

Keep prefill and decode asymmetric

LayerSplit changes prefill ownership. Decode still needs the complete request cache in its ordinary local layout. The pinned conn.py carries enable_dsa_cache_layer_split in prefill registration. When the decode side sees a LayerSplit prefill with a larger CP group, it retrieves fragments from all relevant prefill CP ranks instead of assuming rank zero has the whole cache.

This creates a three-part handoff contract:

1. Every prefill CP rank must register with the same bootstrap room and consistent topology fields.
2. Each rank must expose the fragment for the layers it owns, with indices and pages corresponding to the same request.
3. Decode must wait for the required set, place each fragment into its ordinary full-cache layout, and begin generation only after the handoff is complete.

Do not enable LayerSplit on decode “for symmetry.” The asymmetry is the design. Do not infer that a successful prefill response proves all fragments arrived; inspect bootstrap registration, required response counts, transfer completion, and decode-side cache acceptance.

The pinned prefill.py is also relevant because request completion and transfer timing cross process boundaries. A correct service metric begins when the router admits the request and ends when decode produces the expected first token or complete response. Measuring only the prefill kernel can hide registration, queueing, transfer, and reconstruction costs.

Before load testing, deliberately withhold one prefill CP rank in an isolated environment. The request must not silently continue with an incomplete cache. Then restore the rank and verify clean registration without duplicating a fragment. Ambiguous retry semantics are a stop condition, not an invitation to send the same request repeatedly.

Preflight a pinned launch

Do not treat the following as a copy-and-run production recipe. It is a reviewed flag skeleton showing which side receives LayerSplit. Fill ports, addresses, model paths, TP/DP layout, memory limits, and router settings from the pinned SGLang PD guide for the exact cluster.

Prefill-side LayerSplit flag skeleton

python -m sglang.launch_server \
  --model-path zai-org/GLM-5.2-FP8 \
  --revision ba978f7d347eaf65d22f1a86833408afdb953541 \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend mooncake \
  --enable-prefill-cp \
  --attn-cp-size 4 \
  --cp-strategy interleave \
  --enable-dsa-cache-layer-split \
  --pipeline-parallel-size 1 \
  ... reviewed cluster-specific arguments ...

The decode skeleton deliberately omits the flag:

Decode-side role boundary

python -m sglang.launch_server \
  --model-path zai-org/GLM-5.2-FP8 \
  --revision ba978f7d347eaf65d22f1a86833408afdb953541 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mooncake \
  ... reviewed cluster-specific arguments ...

Add an argument-dump gate before the processes become routable. It should assert the model architecture, role, transfer backend after normalization, prefill-CP boolean, CP strategy, attention CP size, PP size, draft-worker status, model revision, and SGLang commit. Fail if an option is missing rather than relying on a default that may change.

Run these negative controls against the pinned image:

Control	Expected observation
Add LayerSplit to unified role	Startup rejection
Add LayerSplit to decode role	Startup rejection
Change interleave to zigzag	Startup rejection
Change Mooncake to NIXL or Mori	Startup rejection
Set PP2	Startup rejection
Set CP1	Server may start, but logs and allocation must show ordinary non-sharded semantics
Use a non-DSA tiny model	Startup rejection or inactive feature predicate

CP1 is the important soft negative. A process exit code alone cannot confirm that LayerSplit activated. Require resolved-state and pool-size evidence so a mistyped CP setting cannot produce a false “successful optimization” report.

Pinning the package version string is not enough when building from source. Archive the full commit SHA and image digest. The v0.5.18 tag resolved to 71de97b264b04dcd514cf904003028aefe9775c8 for this audit. Re-run the source and test review when either value changes.

Read registered tests without overclaiming

Three registered tests make the upstream intent unusually inspectable:

Registered test	Declared environment	What its presence covers	What it does not prove here
CPU partition and scratch utilities	base-a-test-cpu, estimated 1 s	Balanced ranges, owner lookup, scratch sizing and control behavior	GPU collectives or PD transfer
Owner-broadcast integration	base-c / 4-gpu-b200, estimated 120 s	Four-GPU owner broadcast and remote reads	Your driver, network, image, load, or decode handoff
GLM-5.2 PD+MTP GSM8K	base-c / 8-gpu-b300, estimated 750 s	Intended GLM-5.2 PD topology, MTP coexistence, 1,319 questions, 20 shots, 0.935 floor	That the job passed at audit time or that production quality is unchanged

The files are pinned as CPU utilities, owner broadcast, and GLM-5.2 PD+MTP end to end.

Registration metadata is not a CI result. This audit did not query a green job for the exact commit, run a GPU test, download GLM-5.2, or reproduce the accuracy floor. Even a green upstream job would show one managed environment, not compatibility with another GPU type or a customized launcher.

Use the tests as a blueprint for local evidence. Start with CPU plan checks, then a synthetic multi-GPU broadcast that tags every layer with an unmistakable owner/rank pattern, then a PD request handoff, then a fixed answer corpus. The synthetic pattern is valuable because stale scratch data can be detected without depending on model quality: every received layer should contain the owner and layer identifiers expected by the plan.

For answer parity, preserve prompts, tokenization, sampling controls, seeds where meaningful, max output, stop conditions, request order, and decoding mode. Compare LayerSplit off and on. When exact token equality is not expected under the chosen kernels, define a justified quality metric before seeing the result; do not lower a threshold after a failure.

Build a live canary matrix

One large run is weaker evidence than a staged matrix. Use the smallest production-relevant GLM-5.2 topology that can exercise the complete contract:

Stage	Candidate	Control	Promotion condition
0: argument validation	CP4 LayerSplit prefill	Five invalid flag combinations	Positive resolves exactly; invalid profiles fail as expected
1: allocation	Empty service after initialization	Same topology, LayerSplit off	Every rank logs the expected range and named cache-pool delta
2: owner broadcast	Tagged synthetic layer data	Local-owner reads	Every remote layer matches owner and layer tag; no stale scratch reads
3: PD handoff	One long deterministic request	LayerSplit-off prefill	All ranks register; decode accepts a complete cache once
4: output parity	Fixed prompt corpus	Same pinned image with flag off	Predeclared equality or quality thresholds pass
5: service load	Repeated production-shaped requests	Matched request schedule	TTFT, throughput, failures, and tails meet gates
6: failure and rollback	Rank loss, transfer fault, flag removal	Known-good ordinary cache	Failures are loud and rollback restores service cleanly

Freeze the canary manifest:

Minimum sanitized canary manifest

sglang:
  tag: v0.5.18
  commit: 71de97b264b04dcd514cf904003028aefe9775c8
model:
  id: zai-org/GLM-5.2-FP8
  revision: ba978f7d347eaf65d22f1a86833408afdb953541
topology:
  role: pd-prefill
  attention_cp: 4
  cp_strategy: interleave
  pipeline_parallel: 1
  transfer_backend: mooncake
feature:
  dsa_cache_layer_split: true
expected_ranges:
  - [0, 20]
  - [20, 40]
  - [40, 59]
  - [59, 78]
runtime_evidence:
  model_calls_in_source_audit: 0
  gpu_runs_in_source_audit: 0

Add the container digest, GPU SKU and count, driver, CUDA, Torch, Mooncake version, network devices, page size, cache dtype, model launch args, router revision, and request-corpus hash in the real manifest. Remove credentials, private addresses, request text that contains user data, and raw provider tokens before archiving it.

Repeat enough trials to distinguish a stable shift from startup noise. Keep failed requests in the denominator. Report the number of warmups, measured runs, exclusions, and reasons. A clean median alongside a broken p99 is not a promotion.

Measure correctness, memory, and service impact

Correctness comes first because memory saving is irrelevant if an unowned layer reads stale or misrouted cache data. Instrument these checkpoints:

1. The layer-to-owner plan is identical across every prefill rank.
2. Local and remote lookups return the same values in a tagged synthetic test.
3. Every required prefill rank registers exactly once for a request.
4. Decode receives the full expected fragment set before generation.
5. LayerSplit-on answers pass the predeclared parity or quality threshold.
6. Repeated, concurrent, and long-context requests do not cross-contaminate scratch buffers or bootstrap rooms.

Then measure memory. For every rank, capture pool requested bytes, allocated and reserved device bytes, process peak, whole-device peak, and the phase at which each sample was taken. A CP4 result should be interpreted relative to the source’s 20/20/19/19 ownership and one-scratch sizing, not merely compared to the phrase “about 74%.”

Finally measure the service:

• router-to-first-token TTFT at p50, p95, and p99;
• prefill kernel and owner-broadcast time separately;
• PD transfer and decode-accept time;
• prompt tokens per second and completed requests per second;
• queue time, timeouts, retries, and incomplete fragment sets;
• decode tokens per second to catch an unintended downstream effect;
• usable request/token capacity at a fixed safe memory reserve;
• error rate during steady load, rank restart, and rollback.

Context parallelism itself trades communication for parallel prefill work. The official cookbook warns that its extra all-gather can increase decode latency in unified deployments or hurt short prefill. LayerSplit is restricted to PD prefill, but owner broadcast is another communication event. Use the real prompt-length histogram. A long-context pool may benefit while a short-prompt pool loses.

Write promotion criteria before the run. A reasonable structure is: no correctness regression; no incomplete-transfer errors; a meaningful cache-pool reduction on every rank; total HBM headroom that actually enables the target capacity; and service tails within an explicit budget. Substitute your service numbers, not ours. Public evidence does not provide universal thresholds.

Inject failures and preserve rollback

LayerSplit increases the number of prefill participants whose state contributes to one decode cache. Test failures while traffic is isolated:

Injected event	Required safe behavior
One owner rank stops before broadcast	Request fails or is retried by the reviewed application contract; no stale layer is consumed
One prefill rank never registers	Decode does not construct a partial cache or begin generation
Duplicate registration or delayed response	One fragment is accepted according to an idempotent request identity; ambiguity is surfaced
Mooncake transfer interruption	Request is failed loudly; retry ownership is clear
Mismatched CP size between prefill and decode	Startup or handshake fails before traffic
Model revision differs across roles	Deployment is rejected before cache transfer
Scratch buffer is reused under concurrency	Tagged data and answer parity remain correct
LayerSplit flag is removed	Ordinary full-cache prefill starts and serves the control corpus

Do not use a retry storm to hide an ambiguous handoff. One request may occupy multiple prefill ranks and a decode reservation; an unbounded retry can amplify load while leaving old fragments in flight. Give each request one traceable identity, record its participating ranks, and make the router’s retry policy explicit.

The rollback artifact is the same pinned topology with --enable-dsa-cache-layer-split removed and enough memory for the ordinary full-layer cache. Keep it continuously tested. If the only way to roll back is to reduce capacity or change several parallel dimensions at once, the rollback has not been isolated.

Promotion should stop on any unexplained answer difference, missing rank, duplicate fragment, silent CP1 fallback, unexpected backend normalization, per-rank allocation anomaly, memory OOM in the control, or service-tail regression. Preserve the logs and manifest; a failed canary is useful evidence.

Rent only after fixing the experiment

LayerSplit requires a multi-GPU, communication-heavy canary, so a task-owned rental can be more practical than altering a shared production pool. Rent only after the source revision, model revision, topology, image digest, commands, corpus, metrics, stop conditions, evidence destination, budget, and teardown list are fixed. Confirm the exact GPU count, interconnect, RDMA or TCP path, driver, storage, outbound bandwidth, and whether the chosen shape can run both the LayerSplit candidate and full-cache control.

Run one isolated PD-prefill LayerSplit canary

Use a rental only for the pinned candidate-versus-control matrix. Start with startup negatives and tagged broadcast data, then one PD handoff, a fixed answer corpus, memory telemetry, service distributions, fault injection, and one-flag rollback. Destroy only the resources created for this pilot, and do not transfer a result across SGLang commits, model revisions, GPU types, transfer backends, cache dtypes, or CP layouts.

Check current multi-GPU availability on RunPod →

Affiliate disclosure: This is an affiliate link and a referral link. GLM52.ai may receive a commission or referral credit after qualifying usage at no added charge from us. Renting GPUs does not establish GLM-5.2 DSA cache LayerSplit compatibility, owner-broadcast correctness, Mooncake all-rank transfer, cache-memory savings, output parity, latency, throughput, or production readiness. Verify the live GPU count, topology, interconnect, network mode, image, driver, storage, pricing, egress, and teardown controls before use.

Budget both sides. The candidate may admit more cache capacity, while the control needs enough HBM for all 78 layers on every prefill CP rank. Include model download and storage, image pulls, warmup, repeated trials, log export, failed runs, and idle time. A theoretical percentage is not a reason to leave an expensive cluster running.

Troubleshoot common failures

The process starts but memory does not change

Confirm the resolved attention CP size. CP1 intentionally returns ordinary non-sharded semantics. Then confirm the runner is a non-draft GLM DSA prefill worker, prefill CP is enabled, the strategy is interleave, and the LayerSplit flag reached the resolved args. Compare the named cache-pool allocation phase, not total HBM at two unrelated moments.

Unified or decode startup rejects the flag

That is the intended boundary. LayerSplit belongs only on the PD prefill side. Remove the flag from unified and decode commands. If the service cannot use PD disaggregation, this feature is not an eligible optimization.

NIXL or Mori is required by the cluster

Do not patch out the guard or assume another PD feature’s compatibility transfers. The pinned LayerSplit contract names Mooncake and Mooncake TCP. Keep LayerSplit off or re-audit a later immutable release that adds the required backend with tests.

CP4 logs show ranges other than 20, 20, 19, 19

Stop. Verify that the model has 78 decoder layers, all ranks loaded the same revision, the same CP group formed, and the log uses half-open versus human-readable numbering consistently. Do not continue to a quality test until coverage is exact and non-overlapping.

The cache cell falls but total HBM does not

Weights, graph capture, allocator reserve, communication workspaces, transfer staging, and other pools can dominate. Report both metrics. Decide whether the saved cell increases safe capacity; if not, the optimization may have no operational value.

Output differs only when concurrency rises

Treat that as a correctness failure. Reduce the case to tagged scratch-buffer reuse, overlapping broadcasts, request identity, fragment assembly, and stream synchronization. Do not average the difference into an accuracy score until the race is explained.

Decode waits forever for a fragment

Inspect every prefill CP registration, bootstrap room identity, topology fields, target rank mapping, and transfer completion. Bound the request and surface the missing rank. Avoid automatic repeated submits because an ambiguous first transfer may still be in flight.

MTP works but its cache does not shrink

That matches the pinned helper: draft workers keep a full cache. Measure and label target and draft allocations separately. The GLM-5.2 PD+MTP registered test demonstrates intended coexistence, not symmetric sharding.

Frequently asked questions

Is GLM-5.2 a DSA model?

Yes at the pinned official FP8 revision: its architecture is GlmMoeDsaForCausalLM, and the config exposes DSA index and MLA dimensions. That satisfies the model predicate. The remaining role, CP, backend, and PP constraints still apply.

Does LayerSplit reduce GLM-5.2 weight memory?

No. It shards the DSA GPU KV/indexer cache layers on the PD prefill side. Model weights follow the ordinary serving parallelism. For a separate persistent weight-loading experiment, use the SGLang Weight Cache Daemon audit.

Does it reduce decode-side KV memory?

No in this pinned design. Decode keeps ordinary full-cache semantics and pulls the required fragments from all prefill CP ranks. Enabling LayerSplit on decode is not a supported mirror configuration.

Is the memory reduction always 74%?

No. About 74% is a first-party measurement for one GLM-5.2-FP8 8,192-token CP4 cache cell: 0.77 to 0.20 GB per rank. The source-formula bound is 73.0769% when one scratch layer is included. Neither number is total HBM or a universal forecast.

Should CP8 be preferred because its theoretical bound is larger?

Not automatically. CP8 reduces owned layer equivalents further, but adds more participants and communication. Compare capacity, TTFT, throughput, transfer, failure rate, and tails on the actual prompt distribution.

Can LayerSplit and pipeline parallelism be combined?

Not in the first pilot described here. The pinned constraints require PP1 because CP+PP for this feature is not validated. The separate IndexShare pipeline-split guide solves a different pipeline-boundary problem and does not override this guard.

Does the registered GLM-5.2 test prove production readiness?

No. It shows that upstream intended an eight-GPU PD+MTP quality scenario with specific registration metadata. This audit did not run the job or verify a CI result. Your immutable image and topology need their own execution evidence.

What is the smallest credible promotion packet?

Include resolved startup args, all rank ranges, tagged owner-broadcast parity, all-rank PD registration, fixed-corpus output comparison, named pool and total HBM samples, TTFT/throughput/error distributions, injected-failure outcomes, and a successful one-flag rollback. Include revision and corpus hashes.

Sources and method

The feature introduction is anchored to SGLang v0.5.16 and merged PR #29421. The current audit target is v0.5.18 at immutable commit 71de97b264b04dcd514cf904003028aefe9775c8. Additional feature work is traced through PR #29161 and PR #29166.

Implementation evidence comes from pinned server_args.py, cp/utils.py, dsa_cache_layer_split.py, PD conn.py, and prefill.py. The command context and published boundary come from the pinned GLM-5.2 cookbook.

Test intent is read from the pinned CPU partition test, four-GPU broadcast test, and GLM-5.2 PD+MTP test. Their presence is not represented as an execution result.

Model facts come from the pinned official GLM-5.2-FP8 repository, config, and model card. The fixed first-party discovery routes were the Z.ai GLM-5.2 release, Z.ai LayerSplit discussion, and Zhipu research index. Publisher statements were not treated as independent production verification.

The committed evidence packet records URL, byte count, and SHA-256 for nineteen public receipts; recomputes 78-layer CP2, CP4, and CP8 ranges; reproduces the 74.026% named CP4 reduction; encodes eight configuration outcomes and nine requirements; and preserves the zero-runtime boundary. Its public JSON receipt is byte-identical to the committed result.

No model weights, container, endpoint, grader, GLM, MiniMax, GPU, or NPU were used. This is a dated source and arithmetic audit, not a runtime benchmark. Re-audit implementation, docs, tests, and the model revision after any upgrade.

About the editorial role

Evan Brooks

AI Infrastructure Editor

Evan Brooks serves as GLM52.ai’s disclosed editorial pen name for infrastructure, deployment, and training guides. The role examines checkpoint size, memory math, provider limits, implementation evidence, and operational risk. The GLM52.ai Editorial Team produces and fact-checks every article under this byline.

The GLM52.ai Editorial Team accepts responsibility for this article. Read our editorial policy or request a correction.

An AI image tool created this portrait. It does not depict a real individual.