GLM-5.2 HiCache: L2/L3 Memory and Rollout Audit

Evan Brooks

AI Infrastructure Editor

Editorial responsibility: GLM52.ai Editorial Team · Published and editorially reviewed August 23, 2026

How we research and review · Report a correction

Disclosed editorial pen name · AI-generated portrait

The deployment verdict

Treat GLM-5.2 HiCache as an L2-first, fail-closed pilot. Pin SGLang v0.5.18, leave MTP and L3 off, keep radix caching enabled, choose a fixed host-pool size, and budget it per process/rank. Add the separate DSA indexer allocation: a nominal 120 GB FP8 primary pool calculates to about 144.146 GB per rank, not 120 GB for the node. Promote L2 only after restored-output parity, memory, TTFT, full-request latency, decode throughput, eviction stress, and one-flag rollback pass. Add L3 next. Add MTP last, with acceptance-length gates.

Original editorial illustration: each rank owns an L1-to-L2 path, the amber sidecar represents extra DSA index memory, and the lower mesh represents an optional L3 backend. It is a topology explanation, not a benchmark.

SGLang HiCache extends the GPU radix cache into host RAM and, optionally, distributed storage. For GLM-5.2, that can preserve long reusable prefixes after GPU eviction. It can also allocate far more RAM than an operator expects, restore stale or incomplete state, improve time to first token while making the whole request slower, or damage speculative acceptance if target and draft state do not move together.

The dangerous shortcut is reading --hicache-size 120 as “120 GB for my TP8 server.” At the pinned source, the number sizes the primary host KV pool for one process in decimal GB. Every co-resident rank repeats that allocation. GLM-5.2’s DSA path then creates a separate indexer host pool using the same token-slot count. Page alignment, a 10 GiB free-memory reserve, transfer staging, the model process, and other allocations remain outside the headline number.

This guide pins SGLang v0.5.18 at commit 71de97b264b04dcd514cf904003028aefe9775c8 and GLM-5.2-FP8 revision ba978f7d347eaf65d22f1a86833408afdb953541. Its machine-readable receipt hashes twenty-seven public sources and reproduces four memory profiles. We made zero model calls, zero GPU runs, and zero local upstream-test runs. Source tests and issue measurements are clearly labeled rather than presented as our runtime evidence.

In this guide

1. Separate HiCache from API prompt caching
2. Choose the cache tier before a backend
3. Reproduce GLM-5.2 host-memory geometry
4. Calculate what hicache-size really allocates
5. Do not size from the one-million-token maximum
6. Resolve defaults and compatibility
7. Start with an L2-only control
8. Add L3 only after L2 passes
9. Gate MTP as a separate promotion
10. Measure the whole request
11. Test eviction, corruption, and rollback
12. Rent hardware only after sizing every rank
13. Troubleshoot common failures
14. Frequently asked questions
15. Sources and method

Separate HiCache from API prompt caching

Two features called “cache” solve different operational jobs. The hosted GLM-5.2 prompt-caching guide explains how a provider recognizes a repeated request prefix, reports cached_tokens, and applies a cached-input price. The operator does not allocate provider RAM or choose the storage backend.

HiCache is a self-hosted SGLang memory hierarchy. It manages KV state after GPU-radix-cache eviction:

Feature	Who owns the cache	Primary evidence	Main decision
Hosted prompt caching	API provider	response usage and invoice	prefix stability, privacy, and cost
SGLang L1 radix cache	serving process on GPU	hit/miss and GPU pool telemetry	how much hot KV remains in HBM
HiCache L2	each serving process/rank in host RAM	host slots, bytes, transfer and restore metrics	RAM capacity and CPU↔GPU service impact
HiCache L3	storage service or distributed backend	storage key, prefetch, timeout and error telemetry	cross-instance reuse and failure containment

Neither feature expands the checkpoint’s context architecture. A hosted cache hit can reduce billing without creating local KV state. A HiCache hit can avoid part of a local prefill without changing an API provider’s invoice. Mixing the two intents produces bad sizing, bad observability, and misleading ROI.

The official pinned GLM-5.2 cookbook calls HiCache useful for long-context, prefix-heavy workloads. That is an eligibility clue, not an instruction to enable every tier. First prove that your workload repeatedly reuses prefixes after L1 would evict them. If ordinary GPU radix-cache reuse is already sufficient, L2 transfers add work without a meaningful new hit class.

Choose the cache tier before a backend

HiCache’s hierarchy has three operational boundaries:

• L1: GPU KV cache. Fastest, capacity constrained, and already part of the active SGLang cache path.
• L2: host RAM. Larger and slower. Evicted or selected pages move between GPU and host memory within the server process.
• L3: storage. Optional, potentially shared across instances, and dependent on a file, Mooncake, HF3FS, NIXL, or another registered backend.

The first pilot should answer one narrow question: can L2 recover a reusable GLM-5.2 prefix correctly and improve the chosen service objective without exhausting host RAM? Do not combine that answer with distributed storage, speculative decoding, a new page layout, a new context cap, and a new traffic router in one experiment.

Use this sequence:

1. Run the same pinned model, request corpus, concurrency sweep, context distribution, and acceptance grader with HiCache off.
2. Add L2 only. Keep MTP off and do not set an L3 storage backend.
3. Prove cache-off versus L2 output parity, restored-prefix accounting, memory safety, and full latency.
4. Add one L3 backend while preserving the passing L2 profile.
5. Repeat correctness, service, capacity, timeout, and failure tests.
6. Add MTP only if it has a measurable workload benefit, then inspect target and draft restore state plus acceptance metrics.

This ordering provides a useful fault boundary. If L2 fails, storage is not the cause. If L2 passes and L3 fails, the fault is in storage identity, transfer, prefix synchronization, lifetime, or timeout behavior. If target-only passes and MTP fails, draft-state restoration is the first suspect.

Reproduce GLM-5.2 host-memory geometry

The pinned official config reports model type glm_moe_dsa, 78 decoder layers, one MTP layer, KV LoRA rank 512, RoPE head dimension 64, and DSA index head dimension 128. Those dimensions drive the host-pool stride.

Primary MLA KV bytes

For BF16, each layer-token stores the 512-value KV LoRA vector plus 64 RoPE values at two bytes each:

BF16 primary bytes per layer-token
  = (512 + 64) × 2
  = 1,152 bytes

The pinned DSA FP8 path keeps the no-RoPE vector quantized but stores its scales and the RoPE portion separately. The memory_pool.py comments expose a 528-byte no-RoPE cell—512 data bytes plus sixteen scale bytes—and 128 BF16 RoPE bytes:

FP8 primary bytes per layer-token
  = 512 quantized no-RoPE bytes
  + (512 ÷ 128) × 4 scale bytes
  + 64 × 2 BF16 RoPE bytes
  = 512 + 16 + 128
  = 656 bytes

The extra DSA indexer pool

GLM-5.2’s DSA index is not folded into that primary number. The pinned DSAIndexerPoolHost uses a 128-byte quantized index vector plus a four-byte scale per layer-token:

DSA indexer bytes per layer-token
  = 128 + (128 ÷ 128) × 4
  = 132 bytes

Across the 78 target layers, the combined geometry is:

Primary format	Primary bytes/token	DSA indexer bytes/token	Combined bytes/token	Indexer overhead versus primary
BF16 target	89,856	10,296	100,152	11.458333%
FP8 target	51,168	10,296	61,464	20.121951%
BF16 target + one MTP layer	91,008	10,428	101,436	11.458333%
FP8 target + one MTP layer	51,824	10,428	62,252	20.121951%

The same indexer bytes create a larger percentage surprise with FP8 because the primary pool became smaller while the index dimension did not. “FP8 cache” therefore does not mean all GLM-5.2 cache state costs one byte per dimension.

These are deterministic buffer-cell calculations. They omit page metadata, staging buffers, allocator behavior, Python and storage-client memory, weights, CUDA graphs, request tables, and the operating system. Measure peak resident memory in addition to verifying the calculated named pools.

Calculate what hicache-size really allocates

The pinned pool_host/base.py interprets a positive --hicache-size as decimal GB for the primary host pool. It divides that number by the primary bytes per token, rounds token slots up through the page allocator, and then the DSA registry builds a separate indexer pool with the same slot capacity.

For a 64-token page and 78 target layers, the reproduced totals are:

Primary format	Requested primary GB/rank	Page-aligned primary GB/rank	Extra indexer GB/rank	Combined GB/rank	Combined across 8 co-resident ranks
BF16	64	64.000475	7.333388	71.333863	570.670904
BF16	96	96.003588	11.000411	107.003999	856.031992
BF16	120	120.001610	13.750184	133.751794	1,070.014352
FP8	64	64.001753	12.878402	76.880155	615.041240
FP8	96	96.002630	19.317602	115.320232	922.561856
FP8	120	120.000012	24.146344	144.146356	1,153.170848

The “across eight” column applies when eight rank processes allocate on the same host. If ranks are spread across hosts, budget the relevant subset on each machine. Do not sum it as if all deployments use one NUMA layout; map rank PID, CPU node, pinned memory, and allocation log to the actual host.

For the 120 GB FP8 row, the combined pool is about 134.246756 GiB per rank. The source reserves another 10 GiB in its host-memory availability checks, so the approximate free-memory gate becomes 144.246756 GiB before the allocation attempt. That reserve is headroom, not another advertised cache pool. Staging and ordinary process memory still need space beyond a spreadsheet minimum.

An issue comment provides an important cross-check. In issue #31600, TP0 through TP7 each logged a 120 GB host allocation, followed by an element_size = 656 message. That is consistent with per-rank primary allocation and the FP8 cell geometry. It is not a measured-RSS receipt, and the issue used SGLang v0.5.14, so its numbers are not substituted for current live telemetry.

If you use --hicache-ratio instead, the resolved default is 2.0 outside the decode host-pool retraction special case. A ratio depends on the device pool’s resolved token capacity, so it cannot be converted into host GB from the model config alone. Record the actual device slots and allocation log. A fixed size is easier to compare across canaries, but only after expanding it into per-rank primary, indexer, reserve, staging, and node totals.

Do not size from the one-million-token maximum

The model config exposes a maximum position value of 1,048,576. That value defines an architectural ceiling, not a promise that one request, one server, or one HiCache pool can safely hold the entire window.

The combined target-only cell arithmetic for exactly 1,048,576 tokens is:

Profile	Combined decimal GB per rank	GiB per rank	Decimal GB across TP8
BF16 target, 78 layers	105.016984	97.804688	840.135872
BF16 target + MTP, 79 layers	106.363355	99.058594	850.906840
FP8 target, 78 layers	64.449675	60.023438	515.597400
FP8 target + MTP, 79 layers	65.275953	60.792969	522.207624

These are not launch recommendations. A request also needs active GPU KV, prefill workspaces, attention and DSA intermediate state, model weights, scheduler capacity, and output headroom. Concurrent requests multiply live token pressure, while prefix sharing and eviction change which blocks occupy each tier.

Size from a captured production-shaped distribution instead:

1. record prefix lengths, shared-prefix identities, concurrency, outputs, and accepted-result labels without storing secrets;
2. estimate how many reusable tokens fall out of L1 before their next reuse;
3. choose an L2 capacity that covers a measured hot working set rather than the checkpoint maximum;
4. sweep smaller and larger pools and plot hit rate, transferred bytes, TTFT, full latency, throughput, memory pressure, and failures;
5. stop increasing the pool when marginal accepted-work improvement no longer justifies RAM and transfer cost.

The official HiCache design guide also warns that hit rate is not linear in capacity. More RAM is not a service objective.

Resolve defaults and compatibility

At the pinned revision, the relevant defaults and resolutions are:

Control	Pinned default or resolution	Operational check
HiCache enabled	false	prove the flag is absent in the control and present in the canary
Host-to-device ratio	2.0 when otherwise unset	log resolved device slots and host slots
Fixed size	unset; positive decimal GB overrides ratio	expand per rank and include DSA indexer
Write policy	write_through	first L2-only canary may use write_back to bound transfer load
I/O backend	kernel	verify kernels, fallback warnings, and transferred bytes
Host layout	page_first	record the resolved rather than requested layout
Storage prefetch	timeout	plot completion, partial prefixes, timeout, and fallbacks
L3 threshold	256 tokens	ensure short matches do not look like missing storage

The pinned server_args.py changes a requested page_first layout to page_first_direct when direct I/O is selected. It changes the I/O backend to direct if page_first_direct was paired with kernel. Mooncake also resolves away from layer_first. Save resolved startup arguments and warnings; a configuration file is not proof of the running layout.

Compatibility must be treated as a conjunction:

Proposed combination	Pinned source boundary	Decision
L1 + L2, radix enabled, MTP off	supported baseline shape	first pilot
L1 + L2 + L3, DCP1	supported in principle with a configured backend	second pilot
L3 with DCP greater than one	unsupported	reject
HiCache with radix cache disabled	incompatible	reject
Optimistic prefill, L2, write_back	narrow supported combination	isolate as another experiment
Optimistic prefill with L3 or another write policy	unsupported combination	reject
Direct I/O plus requested page_first	resolves to page_first_direct	accept only if recorded
MTP plus HiCache	merged support plus open related work	add last with draft-state gates

With DCP enabled, the source restricts HiCache to L1/L2 and narrows compatible speculation to MLA/DSPARK cases. It excludes L3, HiSparse, and LMCache in that combination. Do not treat support for each feature separately as proof that the combination is accepted.

Start with an L2-only control

Apply this as a delta to a known-good, immutable GLM-5.2 SGLang launch. It is not a complete model command:

# L2-only canary delta; keep L3 and speculative flags absent.
--page-size 64 \
--enable-hierarchical-cache \
--hicache-size 64 \
--hicache-io-backend kernel \
--hicache-mem-layout page_first \
--hicache-write-policy write_back \
--enable-cache-report \
--enable-metrics

Why start with write_back when write_through is the default? The design guide says write_back moves state to the next tier on eviction and reduces I/O pressure. In an L2-only correctness pilot, that creates an easier first boundary: prove eviction and restore without eagerly copying every access. It is not universally faster. Repeat the workload with the policy that matches the production reuse and durability goal after the base path passes.

Use a fixture with at least four request classes:

1. an exact long-prefix repeat expected to restore from L2;
2. a changed first page expected to miss;
3. a shared prefix with a different final task expected to reuse only the stable portion;
4. enough distinct long prefixes to force L1 eviction, then revisit the first.

For each case, preserve prompt-template hash, request identity, input and output length, expected matched prefix, cache-off output, L2 output, acceptance result, and timing stages. A faster answer that fails the task is not a cache win. A matching answer with unexplained prefix accounting is not ready for storage sharing.

Do not run the canary at one concurrency. Start serially for deterministic parity, then sweep the concurrency and token distributions the service will actually receive. Host transfer races and lifetime bugs often hide under a single request.

Add L3 only after L2 passes

L3 can retain KV beyond one host and share it across compatible instances, depending on backend semantics. It also adds namespace identity, storage availability, serialization, distributed transfer, partial-prefetch, timeout, and cross-rank agreement to the correctness path.

The default timeout policy computes a base two seconds plus 0.1 seconds for each 1,024 tokens, capped at 30 seconds. The reproduced examples are:

Candidate storage prefix	Calculated timeout
8,192 tokens	2.8 seconds
65,536 tokens	8.4 seconds
1,048,576 tokens	30 seconds after cap

Those are defaults, not SLOs. A 30-second allowance can be unacceptable for an interactive route, while a very short timeout can turn L3 into an expensive miss generator. Choose a deadline from the endpoint budget, test partial and complete fetches, and label whether a fallback recomputed or failed.

The design uses cross-rank synchronization to agree on the usable retrieved prefix. That protects the forward pass only if every rank applies the same bounded result. Record the minimum usable prefix, requested storage prefix, bytes received per rank, prefetch duration, and reason for termination.

Test at least these L3 failures:

• storage key absent after metadata says it exists;
• one rank receives fewer pages;
• backend stalls beyond the configured timeout;
• data arrives after the request falls back to prefill;
• an entry is evicted while load-back is in flight;
• two deployments use the same namespace with incompatible model revision, page size, precision, TP mapping, or chat template;
• the backend restarts between write and read;
• an instance detaches storage while requests are active.

PR #31443 merged hybrid/DSA usable-prefix clamping and result synchronization. The reporter in issue #30321 said that work stopped the observed L3 garbling. This is evidence for the canary design and the importance of pinning; it is not a blanket production certificate.

Gate MTP as a separate promotion

GLM-5.2 has an MTP layer and supports speculative serving paths. A restored target cache is insufficient if the draft path lacks corresponding state. The system can show an excellent warm TTFT and then reject most proposals, run extra draft work, or return corrupted output.

Issue #31600 is a useful warning. Its SGLang v0.5.14 reporter measured cold versus store-hit TTFT of 31.5 versus 4.2 seconds, but full duration of 98.99 versus 152.62 seconds. The reported acceptance rate and length also collapsed. Those are old, user-reported measurements from another checkpoint and environment. We do not transfer the magnitudes. We transfer the gate: TTFT alone cannot promote a speculative cache restore.

PR #30393 merged support for packed and sidecar draft caches on August 6, 2026. Related edges were still moving at this audit date:

• PR #32418 remained open to propagate the FP8 override_kv_cache_dim into the legacy draft host pool;
• PR #28896 remained open for DSA draft-indexer restoration;
• PR #31427 remained open to protect host KV until load-back acknowledgement.

The existence of open work does not prove the current unified path is broken. It does make “latest” an unacceptable version specification. Pin the exact container and commit, identify whether the unified or legacy cache path is active, and preserve its startup logs.

Add MTP only after target-only cache restoration passes. Compare cache off, L2 target-only, L2 plus MTP, L3 target-only, and L3 plus MTP. Gate exact or task-level output parity, acceptance rate, mean accepted length, draft tokens, decode tokens per second, TTFT, full latency, and error tails. Roll back MTP independently from HiCache so the target cache can remain available if the speculative branch regresses.

Measure the whole request

One cache experiment needs four evidence layers:

Layer	Minimum measurements	Promotion failure
Correctness	exact/semantic parity, task acceptance, restored prefix, corruption/repetition checks	any unexplained answer or state mismatch
Cache behavior	L1/L2/L3 hits, misses, bytes, pages, evictions, prefetch result, fallback	hit labels inconsistent with restored prefix or data path
Service	TTFT, prefill, decode, end-to-end, throughput, queue, timeout and error distributions	warm TTFT improves but full accepted work regresses
Capacity	primary pool, DSA indexer, staging, RSS, pinned memory, NUMA, reserve and node headroom	allocation exceeds per-rank plan or causes host pressure

Report medians and tails over repeated trials. Separate cold, L1 hit, L2 hit, L3 hit, partial L3, and miss populations. Mixing them into one average hides the path the experiment is supposed to explain.

Use accepted work as the denominator. For an agent workload, record task completion, tool-call validity, loop count, and total tokens. For a deterministic fixture, compare normalized output and the exact cache-off response. When sampling is required, use a task grader and repeated seeds; do not mislabel normal sampling variation as cache corruption.

The official SGLang GLM-5.2 optimization post describes an OpenHands multi-turn workload with an approximately 80K-token initial input, about 220 output tokens per turn, thirteen turns, and roughly 92% aggregate prefix-cache hit rate. That demonstrates why long agent prefixes are interesting. It does not provide HiCache results for this configuration, so none of its throughput values are reused here.

Define a promotion packet before running:

immutable image + SGLang commit + model revision
resolved flags + rank/PID/host/NUMA map
corpus and prompt-template hashes
cache-off, L2, L3, and MTP population labels
correctness + cache + service + capacity distributions
failure-injection outcomes
successful one-flag rollback timestamp

If a metric cannot be tied to a population and configuration, it cannot justify promotion.

Test eviction, corruption, and rollback

Correctness under a warm, unpressured pool is necessary but weak. Force the state transitions that make hierarchical caches useful:

1. Fill L1 with unrelated prefixes until the target prefix is evicted to L2.
2. Restore it repeatedly while adding concurrent misses.
3. Fill L2 until the target is evicted or written to L3 according to policy.
4. Restore from L3 with one delayed rank and one backend timeout.
5. Repeat after process and storage-backend restart.
6. Change the prompt template and model revision to prove namespace isolation.
7. Enable MTP only for the final separate matrix.

Corruption checks should look for more than invalid UTF-8. Compare repeated phrases, tool-call structure, output length, stop reason, task result, and request-to-request leakage. Preserve a minimal failing prefix and state transition without storing sensitive production content.

Rollback should remove --enable-hierarchical-cache and all HiCache-specific arguments, restart the canary pool on the same immutable baseline, and route a small verified traffic slice back. Do not delete shared L3 data during the rollback test; changing storage state makes the recovery evidence harder to interpret. Quarantine a namespace only when the failure specifically requires it and the exact target is known.

Success means the cache-off path serves accepted requests within the rollback objective and host pressure returns to baseline. A config change committed to Git is not rollback evidence until the running process and traffic path confirm it.

Rent hardware only after sizing every rank

The commercial decision follows the memory plan. First decide whether each host can support model weights, GPU topology, the expanded per-rank L2 pools, the DSA indexer, 10 GiB reserve, transfer staging, ordinary service RSS, and a failure margin. Then decide whether L3 requires high-bandwidth local storage, RDMA, or a managed distributed service.

Isolated GPU canary

RunPod can provide an isolated multi-GPU environment for a pinned cache-off versus L2 canary. Before renting, expand –hicache-size per rank, confirm host RAM and NUMA placement for the selected machine, pin the image and model revision, and define the correctness, full-latency, capacity, failure, and rollback gates above.

Check current RunPod GPU availability →

Affiliate disclosure: This is an affiliate link. Public prices and account offers apply; we found no verified link-only buyer discount. GLM52.ai may receive a benefit after qualifying referred usage at no added charge from us. Renting GPUs does not establish GLM-5.2 HiCache compatibility or reproduce this source-only audit.

Do not choose a GPU listing from HBM alone. HiCache’s primary constraint is often host RAM per co-resident process and the bandwidth between host and GPU. Confirm the exact machine, CPU memory, NUMA topology, local storage, networking, container privileges, and availability. A provider listing can change and does not prove SGLang v0.5.18, GLM-5.2 DSA, a storage backend, or MTP passes.

For a broader weights, RAM, VRAM, and storage floor, see running GLM-5.2 locally. For a different DSA cache optimization on a PD-prefill context-parallel topology, use the DSA Cache LayerSplit audit. LayerSplit changes GPU cache-layer ownership; HiCache moves evicted KV across memory tiers. They are not interchangeable toggles.

Troubleshoot common failures

The process requests far more RAM than hicache-size

Confirm whether the log is per rank. Expand the primary pool by 1.114583× for the BF16 DSA indexer or 1.2012195× for the FP8 DSA indexer before page rounding. Then multiply by co-resident rank processes. Add the 10 GiB free-memory reserve, staging, process RSS, and operating-system margin. Check resolved precision and the actual 656-byte versus 1,152-byte primary layer-token cell.

FP8 uses a larger percentage of indexer RAM

The indexer remains 132 bytes per layer-token while the primary cell drops from 1,152 BF16 bytes to 656 FP8 bytes. Its overhead therefore rises from 11.458333% to 20.121951% of primary. This does not mean FP8 uses more combined cache bytes than BF16; it means the unadvertised side pool is a larger fraction.

L2 never hits

Verify radix caching is enabled, the prefix actually reaches L1 eviction, the page-aligned match is long enough, L2 slots exist, and request prefixes are stable from their first token. Check that dynamic attribution, timestamps, tool ordering, or chat-template changes are not inserted before the reusable content. Separate a true L2 miss from an L1 hit that never needed L2.

L3 reports a hit but the output is wrong

Stop promotion. Compare requested, retrieved, synchronized, and usable prefix length on every rank. Inspect model revision, precision, page size, TP mapping, namespace, template identity, late completion, eviction lifetime, and fallback behavior. Reproduce target-only before adding MTP. Do not average corruption into a quality score.

Warm TTFT improves but the request gets slower

Break out storage prefetch, host-to-device restore, draft work, decode, queue, and end-to-end accepted duration. With MTP, inspect acceptance rate and length. Issue #31600 exists precisely because a shorter TTFT can coexist with worse total service. Promote the whole accepted request, not the first token.

Direct I/O is running with the wrong layout

Read resolved startup logs. At the pinned revision, direct plus page_first resolves to page_first_direct; page_first_direct plus kernel resolves the I/O backend to direct. Mooncake also rejects layer_first and selects a compatible page layout. Update the experiment record to the resolved state.

HiCache fails when DCP and storage are enabled

That is an unsupported combination in the pinned source. Keep DCP HiCache at L1/L2 only or remove DCP for the L3 pilot. Do not bypass the compatibility check and describe startup as validation.

Frequently asked questions

What does SGLang HiCache do for GLM-5.2?

It extends the GPU L1 radix cache into a host-RAM L2 pool and optional L3 storage. Reusable KV pages can survive L1 eviction and later be restored. It does not change model weights, make the one-million-token ceiling free, or guarantee a useful hit rate.

Is hicache-size per node or per rank?

At the pinned source, a positive value sizes the primary host KV pool inside each process. In a typical multi-rank launch, each rank allocates its own L2 pool. Sum co-resident ranks on each host, then add the separate DSA indexer and other memory. L3 writeback optimization does not turn L2 into a cluster-wide pool.

Does hicache-size include the GLM-5.2 DSA indexer?

No in this audited allocation path. The fixed decimal-GB value derives token slots for the primary MLA host pool. A separate DSA indexer host pool uses the same slots. Its target-only overhead is 11.458333% of BF16 primary or 20.121951% of FP8 primary before page and staging effects.

Is 120 GB enough for a 120 GB HiCache setting?

No. A target-only FP8 primary request of 120 GB calculates to 144.146356 decimal GB combined per rank after the indexer and page allocation. The availability check also preserves 10 GiB free headroom. The full process needs additional memory.

Should I enable L3 immediately?

No. Prove L2 correctness and service value first. L3 adds storage namespace, distributed transfer, partial-prefix synchronization, timeout, failure, and cross-instance compatibility. It should be a separate promotion with its own rollback.

Which write policy should I use?

The pinned default is write_through. write_back moves data on eviction and can reduce storage pressure; write_through_selective waits for a hit threshold. Start with a policy that isolates the first experiment, then compare on the actual reuse distribution. No policy is universally fastest or safest.

Is MTP safe with HiCache in v0.5.18?

Support for packed and sidecar draft caches merged before v0.5.18, but related legacy FP8, DSA draft-indexer, and lifetime changes remained open at the check date. That does not prove the pinned path is broken. It requires a separate MTP canary with output, acceptance, full-latency, and restore-state evidence.

Do upstream tests prove my deployment works?

No. The pinned source contains a registered DSA host-pool unit test and HiCache variants test. They establish intended test scope. This audit did not run them, and they do not substitute for your image, GPU, topology, backend, workload, or fault path.

Sources and method

The audit target is SGLang v0.5.18 at immutable commit 71de97b264b04dcd514cf904003028aefe9775c8. The GLM-specific entry point is the pinned GLM-5.2 cookbook. Configuration and design boundaries come from the pinned best-practices guide, design guide, and server_args.py.

The memory calculator follows pinned pool_host/base.py, pool_host/mla.py, memory_pool_host.py, and memory_pool.py. Model dimensions come from the official pinned GLM-5.2-FP8 repository, config, and model card.

Historical failure and remediation evidence comes from issue #30321, PR #31443, issue #31600, PR #30393, PR #32418, PR #28896, and PR #31427. Issue numbers, states, comments, and measurements are dated evidence. They are not converted into current benchmark claims.

The fixed first-party discovery checks included the Z.ai GLM-5.2 release and Zhipu AI research index. Neither supplied a narrower HiCache result. The official SGLang optimization post provided workload motivation, not transferable HiCache performance.

The committed receipt records URL, byte count, and SHA-256 for twenty-seven public sources; calculates BF16 and FP8 target/target-plus-MTP profiles; expands 64, 96, and 120 decimal-GB primary pools per rank; multiplies co-resident ranks; and reproduces the default timeout examples. Its public JSON is byte-identical to the committed result.

No model weights, endpoint, container, GLM, MiniMax, GPU, or NPU was used. No upstream test was executed locally. Re-audit source, docs, tests, issues, model revision, and the live machine after any upgrade. Calculated buffer bytes are not measured resident memory or a performance forecast.

About the editorial role

Evan Brooks

AI Infrastructure Editor

Evan Brooks serves as GLM52.ai’s disclosed editorial pen name for infrastructure, deployment, and training guides. The role examines checkpoint size, memory math, provider limits, implementation evidence, and operational risk. The GLM52.ai Editorial Team produces and fact-checks every article under this byline.

The GLM52.ai Editorial Team accepts responsibility for this article. Read our editorial policy or request a correction.

An AI image tool created this portrait. It does not depict a real individual.