GLM-5.2 NVFP4 on SGLang: B200/B300 Setup and Rollout Gates
Independent research — not an official Z.ai publication.Identity and provider disclosure
Original editorial illustration: sparse expert routes are compressed before the deployment branches into eight-GPU and four-GPU canaries. The amber gate represents measured promotion, not automatic production approval.
The useful GLM-5.2 NVFP4 question is not “does four-bit run?” NVIDIA already publishes the checkpoint, and SGLang exposes verified Blackwell recipe cells. The useful question is whether the exact artifact, engine image, topology, and optional acceleration stack pass your workload before they receive production traffic.
This guide pins the NVIDIA checkpoint and three SGLang source files as checked on August 20, 2026. It recomputes the tensor size, compares NVIDIA’s five FP8 and NVFP4 accuracy rows, extracts all nine Blackwell recipes, and preserves the project’s matched B200/B300 benchmark cells. It made zero model calls and zero GPU runs. Project and publisher measurements remain labeled as such. The machine-readable audit receipt contains source hashes, revisions, formulas, results, and limits.
In this guide
Section titled “In this guide”- Choose NVFP4 or keep FP8
- Audit what NVIDIA quantized
- Plan from the 432.874 GiB tensor payload
- Select a supported Blackwell topology
- Launch a conservative SGLang canary
- Add the official performance profile
- Read the project benchmark cells correctly
- Treat publisher accuracy as a test prior
- Gate versions and known regressions
- Run the promotion sequence
- Frequently asked questions
- Sources and method
Choose NVFP4 or keep FP8
Section titled “Choose NVFP4 or keep FP8”NVFP4 is worth a pilot when weight capacity is the binding constraint and the target is a supported NVIDIA Blackwell topology. The artifact reduces the indexed tensor payload from 703.723 GiB for official FP8 to 432.874 GiB. That can create room for cache, graph capture, and higher concurrency, but only the real runtime profile tells you how much room remains.
Keep FP8 when any of these conditions applies:
- the current FP8 deployment already meets cost, latency, and capacity goals;
- the target is H200 or AMD hardware rather than a Blackwell NVFP4 recipe;
- an application has no quantization-specific coding, tool, long-context, and safety evaluation set;
- the team cannot freeze an image digest and checkpoint revision;
- rollback would require rebuilding the service instead of changing a route.
The decision is not “four bits versus eight bits” in the abstract. It is one versioned route versus another, graded on accepted results, peak memory, latency, and failure recovery under the same traffic contract.
| Production condition | First route | Promotion evidence |
|---|---|---|
| Weight capacity is tight on supported Blackwell | NVFP4 canary | Exact revision loads; peak memory and target context fit with margin |
| Current FP8 quality is the accepted baseline | FP8 control plus NVFP4 shadow | No material regression on pre-labeled tasks or parser contracts |
| Low latency is the goal | Plain NVFP4, then official MTP shape | Accept length and end-to-end p95 improve without correctness failures |
| High concurrency is the goal | Plain NVFP4, then DP-attention profile | Sustained throughput improves and health remains responsive at saturation |
| PD disaggregation is required | Single-node first | Separate admission, transfer, structured-output, and recovery evidence |
| Hardware is H20 | Do not infer support | Wait for a merged, released, and reproduced recipe |
Audit what NVIDIA quantized
Section titled “Audit what NVIDIA quantized”The pinned NVIDIA model card describes NVFP4 version 1.0, produced with Model Optimizer 0.46.0. It says the weights and activations of linear operators inside the transformer blocks’ MoE experts are quantized. The shared expert is not quantized.
That boundary explains two operational facts:
- The checkpoint is not a blanket conversion of every tensor to four bits.
- Its byte reduction from FP8 is 38.5%, not 50%.
The pinned config.json still describes GlmMoeDsaForCausalLM: 78 layers,
256 routed experts, eight selected per token, and 1,048,576 positions. NVFP4
does not turn GLM-5.2 into a small dense model, remove DeepSeek Sparse
Attention, or prove that a one-million-token request is affordable.
This is also why active-parameter language is not a capacity plan. Every routable expert must remain available even though only a subset participates in one token. Quantization lowers the weight floor; it does not make the full expert library disappear.
Plan from the 432.874 GiB tensor payload
Section titled “Plan from the 432.874 GiB tensor payload”The pinned safetensors indexes provide a reproducible comparison:
| Checkpoint | Indexed tensor bytes | GiB | Shards | Weight-only floor per rank |
|---|---|---|---|---|
| NVIDIA GLM-5.2 NVFP4 | 464,795,267,072 | 432.874 | 47 | 54.109 GiB at TP8; 108.219 GiB at TP4 |
| Z.AI GLM-5.2 FP8 | 755,617,140,416 | 703.723 | 141 | Not the selected NVFP4 route |
NVFP4 occupies 61.51% of the FP8 tensor bytes and uses 66.7% fewer shards. The shard reduction can simplify staging, but it does not change the total bytes that must cross storage and memory boundaries.
There is an important source discrepancy. The current SGLang configuration contains a comment that a four-GPU GB300 node fits an “~381 GB build.” The pinned NVIDIA index is 464.795 decimal GB, 83.795 GB or 22.0% above that comment. A source-code comment is not an artifact manifest. Use the exact checkpoint index for download and weight-floor planning, then measure the loaded process separately.
The 1,048,576-position config makes context a separate capacity axis. Begin with the production context cap, not the model maximum. Record cache dtype, concurrent sequences, maximum output, graph settings, and peak allocated memory. A successful weight load does not validate a long-context service.
Select a supported Blackwell topology
Section titled “Select a supported Blackwell topology”The pinned SGLang deployment config
maps nvidia/GLM-5.2-NVFP4 to three Blackwell hardware families. Each has
low-latency, balanced, and high-throughput cells marked verified: true.
| Hardware profile | Basic placement | Low-latency | Balanced | High-throughput |
|---|---|---|---|---|
| 8×B200 | TP8 | MTP 5-1-6; chunk 8,192; memory fraction 0.85 | DP8; MTP 2-1-3; chunk 32,768; memory 0.92; max running 256 | DP8; no MTP; chunk 32,768; memory 0.92; max running 512 |
| 8×B300 | TP8 | MTP 5-1-6; chunk 8,192; memory 0.85; graph/running cap 16 | DP8; MTP 2-1-3; decode spec-attention; chunk 8,192; memory 0.85; max running 256 | DP8; no MTP; chunk 8,192; memory 0.85; max running 1,024 |
| 4×GB300 | TP4 | MTP 5-1-6; chunk 8,192; memory 0.85; graph/running cap 16 | DP4; MTP 2-1-3; chunk 8,192; memory 0.92; max running 256 | DP4; no MTP; chunk 8,192; memory 0.92; max running 512 |
The numbers are recipe fields, not defaults to mix freely. For example, the
B300 balanced cell adds --speculative-attention-mode decode, while B200’s
balanced cell uses a larger chunk and higher memory fraction. Copying only the
GPU count and MTP tuple discards part of the validated profile.
H200 appears in the broader GLM-5.2 hardware list for FP8/BF16, not as an NVFP4 cell in this matrix. H20 is absent from the current supported-hardware list. An H20 W4AFP8 recipe PR was still open and unmerged at this audit. Neither route should be relabeled as current GLM-5.2 NVFP4 support.
Launch a conservative SGLang canary
Section titled “Launch a conservative SGLang canary”NVIDIA’s card lists SGLang and vLLM, B200 and B300 test hardware, and a SGLang
launch using modelopt_fp4. It also says the glm_moe_dsa architecture needs
transformers>=5.3.0. Treat its mutable latest image reference as a discovery
starting point, not a production pin.
Build an internal image with an immutable base digest, an exact SGLang commit or release, a pinned compatible Transformers version, and the checkpoint revision. Record the NVIDIA driver, CUDA stack, GPU topology, and any source patch. Then launch the smallest useful canary without optional parallelism:
python3 -m sglang.launch_server \ --model-path nvidia/GLM-5.2-NVFP4 \ --tp 8 \ --quantization modelopt_fp4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code \ --chunked-prefill-size 8192 \ --mem-fraction-static 0.80 \ --host 127.0.0.1 \ --port 8000This is a GLM52.ai canary derivation, not a claim that SGLang benchmarked this exact stripped profile. It deliberately omits EAGLE/MTP, DP-attention, DeepEP, PD disaggregation, and a public bind. Its purpose is to establish checkpoint, parser, kernel, context, and memory health before acceleration adds more failure dimensions.
Do not use trust-remote-code casually. Review the pinned repository code,
stage the model in a restricted environment, and prevent revision drift. If
the service must be reachable beyond loopback, put authentication, TLS,
request limits, and network policy in front of it rather than changing the
bind and calling the deployment complete.
The first receipt should include:
{ "checkpoint": "nvidia/GLM-5.2-NVFP4", "revision": "aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa", "image_digest": "sha256:<record-the-real-digest>", "sglang_revision": "<record-the-real-revision>", "gpu_topology": "8x B200 or 8x B300", "context_cap": 0, "kv_cache_dtype": "<observed>", "mtp": false, "dp_attention": false, "peak_allocated_bytes": 0, "quality_suite_revision": "<pinned-eval-commit>", "rollback_route": "zai-org/GLM-5.2-FP8"}Add the official performance profile
Section titled “Add the official performance profile”Once the plain canary passes, add one layer and repeat the same fixtures.
Add MTP for low latency
Section titled “Add MTP for low latency”Use the official low-latency tuple for the selected hardware:
--speculative-num-steps 5, --speculative-eagle-topk 1, and
--speculative-num-draft-tokens 6. Record accepted length, rejected draft
work, TTFT, inter-token latency, final output, memory, and failures. A higher
accept length is useful only when end-to-end latency improves and the
application result remains accepted.
The B200/B300/GB300 balanced cells use the shorter 2-1-3 tuple. Do not copy the low-latency tuple into a concurrency profile merely because it accepts more draft tokens. Verification overhead can outweigh the saved decode steps.
Our dedicated GLM-5.2 MTP guide covers acceptance measurement and rejected controls. The recipe table here owns deployment selection; that page owns speculative-decoding tuning.
Add DP-attention for concurrency
Section titled “Add DP-attention for concurrency”Balanced and high-throughput recipes set DP equal to the GPU count and enable DP-attention. Apply the hardware-specific chunk size, memory fraction, and request cap together. Saturate with the real prompt and output distribution, not only a fixed synthetic request.
Monitor each rank, not just the aggregate endpoint. A healthy average can hide one rank approaching its cache or graph boundary. Preserve health-check latency, queue depth, resident requests, KV use, failed requests, and restart count through the soak.
Add PD disaggregation last
Section titled “Add PD disaggregation last”PD changes admission, KV transfer, routing, recovery, and grammar/MTP paths. It needs a separate experiment with prefill and decode role receipts. Do not infer it from a successful all-in-one node. Start with one bounded context and one concurrency class; force transfer, worker loss, router timeout, and structured-output cases before expanding.
Read the project benchmark cells correctly
Section titled “Read the project benchmark cells correctly”The pinned SGLang benchmark file publishes B200 and B300 NVFP4 cells for an 8,192-token random input and 1,024-token output. Its per-GPU metric combines input and output throughput; TTFT and time per output token remain separate. GLM52.ai did not rerun them.
| Strategy | Concurrency | B200 project tok/s/GPU | B300 project tok/s/GPU | B200 TTFT | B300 TTFT |
|---|---|---|---|---|---|
| Low latency | 1 | 527 | 459 | 295 ms | 196 ms |
| Low latency | 16 | 2,289 | 2,016 | 2,491 ms | 274 ms |
| Balanced | 64 | 3,770 | 1,377 | 5,837 ms | 680 ms |
| Balanced | 256 | 5,343 | 1,845 | 16,736 ms | 3,010 ms |
| High throughput | 1,024 | 5,305 | 3,870 | 130,174 ms | 6,370 ms |
B200 has higher reported per-GPU throughput at all five matched points; B300 has lower TTFT in the same table. The recipes also differ, and the cookbook notes that current B300 kernels are not optimized like B200’s. Therefore the table does not establish that one GPU is universally faster. It establishes which exact project profiles are worth reproducing for your queueing objective.
Do not select the high-throughput row for an interactive service: its project TTFT is 130.174 seconds on B200 and 6.370 seconds on B300 at concurrency 1,024. Likewise, do not select the lowest TTFT row for a saturated batch service. Define the service-level objective first, then reproduce the closest profile.
Treat publisher accuracy as a test prior
Section titled “Treat publisher accuracy as a test prior”NVIDIA reports the following FP8 baseline and NVFP4 rows. AA-LCR was measured with SGLang; the card says the other four used vLLM. These are publisher results, not a universal equivalence certificate.
| Publisher benchmark | FP8 | NVFP4 | Point change |
|---|---|---|---|
| GPQA Diamond | 89.52 | 89.39 | −0.13 |
| SciCode | 49.85 | 49.04 | −0.81 |
| IFBench | 74.95 | 75.81 | +0.86 |
| AA-LCR | 69.38 | 70.13 | +0.75 |
| τ²-Bench Telecom | 97.90 | 98.25 | +0.35 |
The range is −0.81 to +0.86 points. That is consistent with “small differences on these five harnesses,” not “no quality loss.” Quantization can change a specific codebase, tool schema, language, long prompt, or rare routing pattern without moving these aggregate rows.
Build an application suite with pre-labeled acceptance. Include normal coding, multi-file changes, long retrieval, invalid tool arguments, required tool use, JSON schema, refusal, multilingual prompts, repeated prompts, and the longest context you will serve. Grade final state and tool behavior, not prose style. Compare NVFP4 to the exact FP8 control with the same parser and sampling contract.
Gate versions and known regressions
Section titled “Gate versions and known regressions”Version pinning matters because NVFP4 and DSA paths continued changing after
the checkpoint appeared. SGLang v0.5.17 was the latest release at this audit
and its notes include later DSA and NVFP4 fixes. “Latest” tomorrow may contain
a different kernel, backend selector, graph path, or quantized MoE loader.
Use upstream issues as regression fixtures with precise scope:
| Upstream evidence | State at audit | What to do |
|---|---|---|
| NVFP4 + EAGLE CUDA-graph capture | Open | Test the exact official 5-1-6/2-1-3 shapes on the pinned image; do not generalize a reported 6-1-7 failure or apply an unreviewed source patch |
NVFP4 long-context NaN / ! collapse |
Closed | Keep long, content-varied prompts plus finite-logit/output checks in every image upgrade |
| PD + DP-attention resident-KV freeze | Open; reported on FP8 | Test running plus transferred/resident KV growth and health responsiveness before PD promotion |
| PD + EAGLE + grammar double accept | Closed | Keep required-tool and structured-output fixtures; confirm the fixed path exists in the pinned build |
An open issue is not proof that current main fails every recipe. A closed issue is not a reason to delete its test. The correct response is a versioned canary that can distinguish load failure, graph capture, silent non-finite output, parser corruption, capacity wedge, and ordinary application rejection.
Reject the build on any of these signals:
- repeated punctuation or empty output on a prompt whose FP8 control passes;
- non-finite logits, probabilities, memory metrics, or benchmark values;
- invalid or prematurely executed tool arguments;
- a health endpoint that stalls when one rank approaches its cache boundary;
- output changes after only enabling MTP or DP-attention without a documented acceptance explanation;
- an image, checkpoint, or dependency revision that is missing from the receipt;
- a failed rollback rehearsal.
Run the promotion sequence
Section titled “Run the promotion sequence”Use one promotion ladder. Every step reuses the same quality corpus and adds only the load appropriate to that layer.
- Freeze artifacts. Record checkpoint revision, model index hash, image digest, SGLang revision, Transformers version, driver, CUDA stack, parser flags, and topology.
- Load offline. Verify 47 expected safetensor shards, exact tensor-index hash, no unexpected network retrieval, and clean shutdown.
- Pass plain TP. Run short, long, coding, tool, structured, and adverse inputs without MTP, DP-attention, or PD. Record peak memory and outputs.
- Compare FP8. Use the same tasks, settings, and graders. Investigate route-specific failures rather than averaging them away.
- Enable one optimization. Add the selected MTP or DP profile. Measure acceptance, TTFT, output rate, memory, and error class.
- Soak at the declared cap. Include mixed prompt lengths, maximum output, cancellation, queue pressure, and worker restart.
- Rehearse rollback. Route back to FP8 while requests are in flight; verify idempotency and preserve receipts.
- Promote a small traffic slice. Set an automatic stop on severe quality, tool, memory, or availability failures. Expand only after the observation window closes cleanly.
The final comparison metric is cost or capacity per accepted result, not raw token throughput. Include failed attempts, retries, queue time, accelerator hours, storage, data transfer, and review. NVFP4 succeeds when the saved capacity creates operational value without crossing a correctness or recovery gate.
Frequently asked questions
Section titled “Frequently asked questions”Is GLM-5.2 NVFP4 a single-GPU model?
Section titled “Is GLM-5.2 NVFP4 a single-GPU model?”No. Its indexed tensors occupy 432.874 GiB before runtime and cache. Current SGLang NVFP4 cells use 8×B200, 8×B300, or 4×GB300. Dividing weights by ranks is only a floor, not proof that a smaller unsupported topology works.
Does NVFP4 halve GLM-5.2 FP8 memory?
Section titled “Does NVFP4 halve GLM-5.2 FP8 memory?”No. The indexed tensor payload is 38.5% smaller. NVIDIA says the MoE experts’ linear weights and activations are quantized while the shared expert remains unquantized. Peak process memory needs a real run.
Is B300 always faster than B200 for this checkpoint?
Section titled “Is B300 always faster than B200 for this checkpoint?”The pinned project cells do not support that statement. B200 reports higher per-GPU combined throughput at the five matched points; B300 reports lower TTFT. Recipes and current kernel maturity differ. Reproduce the profile tied to your service objective.
Can I use the current H20 W4AFP8 pull request?
Section titled “Can I use the current H20 W4AFP8 pull request?”Not as a released NVFP4 recipe. It remained open and unmerged, and H20 was absent from the current supported-hardware list. A research branch may be tested in isolation, but it must not inherit the verified label.
Should MTP be enabled on the first launch?
Section titled “Should MTP be enabled on the first launch?”No. Establish a plain checkpoint and parser baseline first. Then add the hardware-specific 5-1-6 or 2-1-3 recipe and compare the same fixtures. This makes graph, draft, and acceptance regressions attributable.
Does NVIDIA also support vLLM?
Section titled “Does NVIDIA also support vLLM?”The pinned model card lists SGLang and vLLM and supplies commands for both. This page audits the current SGLang recipes. Do not copy SGLang backend, parallelism, or issue conclusions into vLLM without a separate pinned audit.
Sources and method
Section titled “Sources and method”The proactive research route started from the current SGLang GLM-5.2 cookbook, its pinned deployment cells, the NVIDIA NVFP4 checkpoint, the official Z.AI GLM-5.2 release, and Zhipu’s research index. The current AI HOT batch had no item with a direct, non-duplicate GLM-5.2 reader task, so no feed summary was used as factual evidence or forced into the article.
The committed audit hashes ten source snapshots. Its calculation script reads the checkpoint indexes and hand-audited source fields, then derives byte ratios, per-rank weight floors, accuracy changes, and matched project rows. Every figure in the article can be checked in the public receipt. No checkpoint shards were downloaded, no SGLang server was started, and no throughput or accuracy result is represented as GLM52.ai hardware evidence.