Evaluate GLM-5.2 with Inspect AI

Independent research — not an official Z.ai publication.Identity and provider disclosure

AI Infrastructure Editor

Editorial responsibility: GLM52.ai Editorial TeamPublished and editorially reviewed August 18, 2026

How we research and review Report a correction

Disclosed editorial pen name · AI-generated portrait

A cold-blue evaluation pipeline routes three test cases through a GLM-5.2 node, a verified API request, stacked audit logs, and three scoring gates

Original editorial diagram of the evaluation boundary: task cases enter one model route, the API envelope is verified, and only then do logs and scores become evidence. It contains no real dashboard data.

Use openai-api/zai/glm-5.2, set ZAI_API_KEY and ZAI_BASE_URL=https://api.z.ai/api/paas/v4, and keep the Responses API off until Z.AI documents that route. Start with one deterministic task, one connection, a small output cap and no automatic retry. Inspect the first API envelope and eval log before expanding the suite.

That configuration follows Inspect’s documented OpenAI-compatible provider convention and Z.AI’s documented general API endpoint. Our no-network audit observed all three requests at /api/paas/v4/chat/completions, preserved model: glm-5.2, applied bearer authentication, forwarded max_tokens: 16 and temperature: 0, recovered three exact completions, and finished the log with status success.

Follow the GLM-5.2 Inspect AI workflow

Choose Inspect for the right evaluation job
Freeze the route and package versions
Install a pinned environment
Build the smallest auditable task
Run one bounded evaluation
Verify the request before trusting the score
Read the eval log as evidence
Design a scorer that matches the contract
Graduate from fixture to live GLM-5.2
Diagnose failures by layer
Audit the dated sources and limits
Resolve common Inspect AI questions

Choose Inspect for the right evaluation job

Inspect AI is an evaluation framework from the UK AI Security Institute. A task combines a dataset, a solver that elicits model output, and one or more scorers. Each run writes an eval log that keeps task configuration, model events, samples, output and scores available for inspection.

That is useful when the question is not “is GLM-5.2 good?” but a smaller, testable contract:

Production risk	Inspect task	Observable pass condition
A route silently points at the wrong API	one marker prompt	the captured path, model and response shape match the frozen profile
A parser accepts extra prose	one exact or schema-bound sample	the scorer rejects output outside the declared contract
A prompt revision changes behavior	a pinned dataset with stable sample IDs	the new log can be compared with the prior revision
A long agent trace fails	a task with tools, limits and event inspection	the log separates transport, generation, tool and scoring failures

Do not begin with a giant benchmark. A hundred samples through the wrong base URL create a precise-looking answer to the wrong question. First prove that the model route, request fields, task revision and scorer are the ones you intended to test.

This page is deliberately distinct from the GLM-5.2 smevals workflow. That guide owns a small file-based runner, two real API receipts and offline rubric regrading. This guide owns Inspect’s provider adapter, Python task composition, model-event inspection and eval-log boundary.

Freeze the route and package versions

The checked profile is:

Layer	Pinned value	Why it matters
Inspect AI	`0.3.249`	freezes task, model-adapter, scorer and log behavior
OpenAI client	`2.48.0`	freezes the compatible HTTP client used by the adapter
Inspect model name	`openai-api/zai/glm-5.2`	maps the provider label `zai` to scoped environment variables
API key variable	`ZAI_API_KEY`	follows Inspect’s uppercased provider convention
Base URL variable	`ZAI_BASE_URL`	prevents the request from falling back to another provider host
Z.AI general base	`https://api.z.ai/api/paas/v4`	selects the documented metered API, not Coding Plan
API family	Chat Completions	matches Z.AI’s published `/chat/completions` example
Responses API	`false`	prevents an unverified API-family switch

Inspect documents model names as openai-api/<provider-name>/<model-name>. It derives the variables <PROVIDER_NAME>_API_KEY and <PROVIDER_NAME>_BASE_URL; hyphens in the provider name become underscores. Therefore zai resolves to the two ZAI_* variables above. Calling the model openai/glm-5.2 would select a different provider adapter and credential namespace.

Z.AI publishes https://api.z.ai/api/paas/v4 as its general endpoint and shows glm-5.2 at /chat/completions with bearer authentication. Its GLM-5.2 model guide describes a text-in, text-out model with a one-million-token context and several reasoning controls. Those capabilities do not make every Inspect feature automatically compatible; test tools, structured output, streaming, reasoning fields and long context as separate profiles.

Install a pinned environment

Use an isolated virtual environment and record the interpreter as well as the package versions:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install "inspect-ai==0.3.249" "openai==2.48.0"
python - <<'PY'
from importlib.metadata import version
print("inspect-ai", version("inspect-ai"))
print("openai", version("openai"))
PY

The dated PyPI metadata requires Python 3.10 or newer for this Inspect release. Our fixture used Python 3.12.11 inside the digest-pinned image recorded in the sanitized result JSON.

Pinning is not busywork. A change in the provider adapter can alter endpoint selection, request translation, default retry behavior or log fields even when the task file does not change. Preserve a lock file or image digest with every run you may later compare.

Build the smallest auditable task

Create one sample whose result can be checked without another model:

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import exact
from inspect_ai.solver import generate


@task
def glm52_route_check():
    return Task(
        dataset=[
            Sample(
                id="route-marker",
                input="Return exactly: ROUTE-17",
                target="ROUTE-17",
            )
        ],
        solver=[generate()],
        scorer=exact(),
        name="glm52_route_check",
        version="2026-08-18",
    )

The sample ID distinguishes the case in logs. The target is deterministic. generate() uses the evaluation’s selected model, while exact() compares the completed output with the target. This checks the full dataset → solver → model → output → scorer path without pretending that one marker measures coding, reasoning or agent ability.

For a production task, replace the marker only after defining one observable failure. A JSON contract can use a deterministic custom scorer; a factual task can compare against a frozen answer set; a tool task can inspect the tool-call arguments and environment outcome. The structured-output guide covers the separate distinction between JSON mode, schema enforcement and application validation.

Run one bounded evaluation

Keep credentials outside the task file:

export ZAI_API_KEY="your-Z.AI-api-key"
export ZAI_BASE_URL="https://api.z.ai/api/paas/v4"

Then run one sample with the Chat Completions family explicit:

inspect eval glm52_route_eval.py \
  --model openai-api/zai/glm-5.2 \
  -M responses_api=false \
  --max-tokens 16 \
  --temperature 0 \
  --max-connections 1 \
  --max-retries 0 \
  --log-model-api

The low token cap matches the marker job. One connection avoids an accidental burst, and zero automatic retries keeps an ambiguous submission from quietly becoming another billable attempt. --log-model-api makes the first request and response inspectable, but it also makes the log more sensitive. Store the log as application data, not as a public build artifact.

The general metered API and Coding Plan are different products. Do not infer that a generic evaluation SDK is entitled to subscription quota merely because the payload is OpenAI-compatible. The Coding Plan versus API guide explains that access boundary.

Verify the request before trusting the score

An exact score can be correct while the experiment is wrong. Inspect the first model event and confirm all of these before reading the aggregate:

Envelope field	Required observation	Reject when
destination	official HTTPS Z.AI host and `/api/paas/v4/chat/completions`	base URL is absent, doubled, redirected unexpectedly or points elsewhere
model	exactly `glm-5.2`	a gateway alias or fallback changed the tested model
API family	Chat Completions	a Responses request appears without a documented Z.AI contract
messages	the expected role and marker prompt	a system layer, template or prior history was inserted unexpectedly
generation cap	16 for this marker	a client translated or dropped the bounded cap
sampling	temperature 0 for this deterministic check	the actual request differs from the frozen profile
authentication	bearer scheme applied	credentials are missing, copied into the body or exposed in an artifact

Our isolated fixture observed three requests with this exact shape:

Sample	Path	Model	Cap	Completion	Exact score
`mint`	`/api/paas/v4/chat/completions`	`glm-5.2`	16	`MINT-08`	correct (`C`)
`orbit`	`/api/paas/v4/chat/completions`	`glm-5.2`	16	`ORBIT-17`	correct (`C`)
`pine`	`/api/paas/v4/chat/completions`	`glm-5.2`	16	`PINE-42`	correct (`C`)

The fixture server listened only on loopback inside a container with --network none. It returned deterministic OpenAI-compatible envelopes, not model generations. The C is Inspect’s categorical value for correct; it is not a letter grade or a claim that GLM-5.2 scored “C.”

Read the eval log as evidence

Inspect writes one eval log per evaluated task. The current default .eval format is optimized for size and incremental access; use the log API or viewer instead of treating it as hand-edited JSON.

from inspect_ai.log import list_eval_logs, read_eval_log

log_info = list_eval_logs("logs", descending=True)[0]
log = read_eval_log(log_info)

assert log.status == "success"
assert log.eval.model == "openai-api/zai/glm-5.2"
assert len(log.samples or []) == 1

sample = log.samples[0]
print(sample.id, sample.output.stop_reason, sample.scores)

Run inspect view when you need the message history, model events, scoring decision and metadata together. The official eval-log documentation notes that Inspect logs raw model API requests and responses for the first few calls by default when API logging is enabled. That is excellent for route verification and a reason to protect logs containing private prompts, model outputs or tool results.

Preserve at least these fields with a release decision:

task name and version, sample ID and dataset revision;
Inspect, client and Python versions;
requested model, base host, API family and generation profile;
start time, terminal status, error class and retry count;
output, finish reason, score and scorer revision;
usage and provider request identity when the live route returns them;
a digest of sensitive raw logs when the logs cannot be published.

Do not publish API keys, authorization headers, private prompts, hidden reasoning, customer content or full provider responses. A sanitized summary should state exactly which fields were omitted.

Design a scorer that matches the contract

Scoring turns model output into an observable decision; it does not repair a vague task. Choose the least subjective scorer that actually matches the job:

Output contract	First scorer to consider	Required negative control
exact marker or enum	`exact()` or a small deterministic matcher	extra prose must fail
JSON object	parse, validate exact keys and compare required values	malformed JSON and extra keys must fail
numeric result	numeric normalization with declared tolerance	a plausible but out-of-tolerance result must fail
tool call	validate tool name, argument schema and environment outcome	wrong tool and unauthorized argument must fail
open-ended answer	rubric plus calibrated grader or human review	known bad and borderline answers must expose grader error

Inspect’s scoring documentation separates raw output, scores and aggregate metrics. Keep that separation in your reporting. A transport error is not a zero-quality answer; a parsed answer that fails a rubric is not a runner error; and a changed scorer can alter the score without the model output changing.

Run positive and negative fixtures before spending on model calls. A scorer that marks both correct cannot validate a live run. For a model-graded scorer, pin the grader model separately, record its prompts and estimate agreement against known labels before using it as a release gate.

Graduate from fixture to live GLM-5.2

The local route test removes one class of uncertainty. It does not authorize a provider call or a larger evaluation. Before the first live sample:

confirm the account, region, current price, retention policy and rate limit;
calculate the maximum prompt plus output exposure for one request;
use a synthetic, non-sensitive prompt and a deterministic target;
set one connection, a short bounded timeout, zero automatic retries and an explicit output cap;
verify the rendered request and response event before increasing sample count;
preserve the provider’s usage and request receipt without the credential;
stop on an ambiguous timeout, unexpected redirect, wrong API family, unknown model or unrecognized response shape.

Once the single sample passes, grow deliberately: add a known-negative sample, repeat enough epochs to measure variability, then add production-shaped cases. Set a total request and token budget before parallelism. A million-token model context is a capacity claim, not a recommendation to start an eval at that size.

For self-hosted evaluation, point the same openai-api provider pattern at a separately secured OpenAI-compatible server and change both the provider label and model ID to match that server. Do not reuse zai for a local endpoint; the name should keep credentials and receipts attributable to the actual route.

Diagnose Inspect AI failures by layer

Symptom	Evidence to inspect first	Likely layer	Correct response
missing `ZAI_API_KEY` error	selected provider name and environment	configuration	use `openai-api/zai/...`; do not rename the variable to `OPENAI_API_KEY`
404 or doubled path	base URL and captured destination	route construction	set the base to `/api/paas/v4`, not the full `/chat/completions` request path
401 or 403	account, bearer application and provider error class	authorization	stop; verify account and key scope without printing the key
request reaches `/responses`	model args and API event	API-family mismatch	set `responses_api=false`; do not retry until the route is documented
run status `error` with no score	transport exception and model event	provider or harness	fix the request layer; do not report a model-quality failure
run succeeds but exact score fails	completion, whitespace and target	output contract	inspect the actual output; revise prompt or scorer transparently
every model fails the same negative control	task and scorer fixture	evaluation bug	repair the task before comparing models
repeated calls after a timeout	retry setting and provider request IDs	ambiguity handling	stop automatic replacement; reconcile usage before retrying

If tools are added, Inspect sets strict tool schemas by default for the OpenAI-compatible provider and offers controls for emulation and strictness. Do not disable strictness merely to make an error disappear. First compare the exact Z.AI function-calling contract, then run one harmless tool fixture. The GLM-5.2 tool-calling guide covers the full assistant-tool-assistant loop and route-specific compatibility boundaries.

Audit the dated sources and limits

Checked on August 18, 2026 (Hong Kong time). Sixteen public receipts cover Inspect providers, tasks, logs, scoring, the pinned 0.3.249 repository and PyPI release; OpenAI 2.48.0; Z.AI’s API introduction, GLM-5.2 guide, Chat Completion reference and release page; the Hugging Face model API and pinned model card; the Zhipu research index; and the exact discovery pages.

The topic originated from AI HOT’s Inspect AI item, which points to a Google AI DEV article. Those pages are discovery attribution, not evidence that Inspect supports GLM-5.2. The provider contract comes from Inspect; the endpoint and model contract come from Z.AI and the pinned model card.

One exact Google query, GLM-5.2 Inspect AI evaluation, returned eight organic results but no result that joined the provider name, Z.AI environment pair, Chat Completions request evidence, eval log and scoring boundary. SerpAPI request serpapi-ffa1bc559e5d4e24a33b7251b57721fe was evaluated high-value because it confirmed this narrow supply gap.

The reproducible source audit stores hashes and HTTP receipts but no response bodies. The protocol script ran three fixture samples in a digest-pinned Python container with networking disabled. All 7 public-contract assertions and 13 protocol assertions passed. The result contains zero authenticated requests, zero real model calls, zero downloaded weights and no credential.

This evidence cannot establish live endpoint availability, model behavior, tokenization, latency, billing, retention, tool calling, structured output, streaming, sandboxing, grader validity or future compatibility. Recheck the current contracts and run an authorized bounded smoke test before using this profile as release evidence.

GLM-5.2 Inspect AI FAQ

What Inspect AI model name should I use for Z.AI GLM-5.2?

Use openai-api/zai/glm-5.2. Under Inspect’s provider convention, zai selects ZAI_API_KEY and ZAI_BASE_URL; glm-5.2 remains the model sent in the request body. Verify the first model event rather than trusting the string alone.

What should ZAI_BASE_URL contain?

For Z.AI’s general metered API, set https://api.z.ai/api/paas/v4. Do not append /chat/completions to the base variable; the compatible client adds that request path. Coding Plan has a separate endpoint and product boundary.

Should I enable the Responses API?

Not for this verified profile. Inspect can opt into the Responses API, but the Z.AI contract checked here publishes Chat Completions for glm-5.2. Keep responses_api=false until the exact endpoint documents and passes a separate Responses fixture.

Does a successful exact score prove GLM-5.2 is good?

No. It proves that one output met one scorer. In our no-network fixture it also proves only that Inspect parsed and scored a deterministic compatible response. Model-quality claims require authorized live samples, a task-valid dataset, negative controls, repeated measurements and uncertainty reporting.

Are Inspect eval logs safe to publish?

Not by default. They can contain prompts, outputs, tool results, metadata and raw API request/response events. Keep raw logs private, remove secrets and personal data, and publish a sanitized receipt that states exactly what was omitted.

How is this different from smevals?

This workflow focuses on Inspect’s Python task components, compatible-provider adapter and rich eval logs. The smevals guide focuses on a small external runner, inspectable task files, stored outputs and post-hoc regrading. Choose the workflow whose artifacts match the release decision; do not combine scores from different harnesses without validating both contracts.