GLM-5.2 Factual Recall: Build a WikiProfile Test

Independent research — not an official Z.ai publication.Identity and provider disclosure

Model Research & Evaluation Editor

Editorial responsibility: GLM52.ai Editorial TeamPublished and editorially reviewed August 13, 2026

How we research and review Report a correction

Disclosed editorial pen name · AI-generated portrait

Fact library feeding a retrieval key and paired direct and reverse questions into a GLM-5.2 audit gate

Original editorial illustration of the test design. The shelf represents the pinned fact set, the key represents access to encoded knowledge, and the paired cards represent direct and reverse questions. It is not a model dashboard or a reported GLM-5.2 result.

A wrong factual answer does not tell you whether GLM-5.2 never encoded the fact or failed to retrieve knowledge it can expose under another prompt. Treating both cases as “hallucination” hides the engineering decision. Missing knowledge may need retrieval or a different model. Unstable access may need a better prompt, thinking policy, verification step, or escalation route.

Google Research’s WikiProfile work supplies a useful diagnostic pattern: probe one fact through completion, direct questions, reverse questions, natural paraphrases, multiple choice, context, and thinking. The associated paper reports results for 13 models, including a model named GPT-5.2. That is OpenAI’s model: GPT-5.2 is not Z.ai GLM-5.2. This page transfers the method, not the published score.

Our independent preflight pins the public dataset at revision 0448b6abf3d6fb0a964e6935c0265dbd47584cda, verifies every row, and freezes a small pilot before any model can influence sample selection. The output is a test plan you can run later—not a favorable result invented from a paper about another model. The evidence receipt records zero model calls.

Separate encoding from recall

WikiProfile treats the fact, not an isolated question, as the unit of analysis. Its tasks ask whether a model can expose the same proposition across several access paths. That creates a more useful fault tree than one accuracy percentage.

Observed pattern	Careful interpretation	Engineering response
Completion and contextual tasks fail	Possible encoding failure; the fact may not be available from model parameters	Retrieve a trusted source, change the task, or test another model
Completion/context pass, closed-book questions fail	Possible recall failure; access is unstable despite encoding evidence	Try a frozen thinking condition, phrasing controls, or retrieval
Direct passes, reverse fails	Directional access failure	Add reverse probes and avoid treating one familiar phrasing as coverage
Free response fails, multiple choice passes	Recognition exceeds generation	Do not convert recognition into a claim that the model can produce the fact reliably
Thinking recovers an encoded fact	Thinking-assisted recall for that fact and setup	Measure added tokens, delay, and whether the recovery repeats
Every format passes once	One successful sample	Repeat under the same route before calling it reliable

These labels are behavioral. A completion success does not reveal a neuron or prove a fact is stored in one identifiable location. A failure does not reveal the model’s training corpus. The method is valuable because it separates observable access patterns while keeping mechanistic claims out of the result.

The distinction also changes a production fallback. If a rare customer-policy fact is not encoded, asking the model to “think harder” may increase cost without supplying missing evidence. If the fact is recoverable but unstable, a verified retrieval result may still be safer than spending variable reasoning tokens. Diagnose first; route second.

Do not transfer GPT-5.2 results to GLM-5.2

The paper evaluates Gemini, GPT, and Gemma families. Its model list includes GPT-5.2 and contains no GLM-5.2 row. The paper’s finding that recall accounts for a large share of GPT-5.2 errors therefore supports a hypothesis to test on GLM-5.2, not a GLM result.

Label in a source	Organization	Can it support a GLM-5.2 score?
GPT-5.2	OpenAI	No
GLM-5.2	Z.ai / Zhipu	Only with GLM-specific runs
WikiProfile aggregate	Google Research paper	Method and dataset context only
This article’s 32 assertions	GLM52.ai data audit	Dataset integrity and pilot design only

The decimal is not a shared model lineage. Similar names also do not align tokenizer, training data, reasoning mode, provider, sampling policy, release date, or grading behavior. Search snippets that omit the first letter make this an easy attribution error, so record the full provider and model ID in every run.

Z.ai’s GLM-5.2 release page describes the model and its official evaluation claims. It does not claim a WikiProfile result. The official checkpoint card is the right source for checkpoint identity, but it also contains no WikiProfile run in the evidence reviewed here.

Audit the pinned WikiProfile data

The public WikiProfile dataset card documents 2,150 facts, ten question instances per fact, Wikipedia-derived source text, generated annotations, search-grounded filtering, manual validation, and a CC-BY-SA 4.0 license. Our preflight downloads the immutable CSV and card by revision, then fails if either SHA-256 changes.

Audited field	Observed value	Why it matters
CSV bytes	7,502,273	Pins the complete downloadable artifact
Unique `fact_id` rows	2,150	Prevents duplicate facts from inflating a run
CSV columns	23	Detects a silent schema change
Documented task instances	21,500	Ten formats per fact in the dataset design
Distinct `page_id` values	1,883	Observed source-page identity in the CSV
Named pilot categories	9	Supports one low/high pair per named topic
Rows labeled `Other`	1	Preserved, but excluded from the nine-topic pilot
Direct-answer equality	2,150 / 2,150	Every direct gold equals the object field
Reverse-answer equality	2,150 / 2,150	Every reverse gold equals the subject field

The sanitized machine-readable receipt contains the category counts, popularity cutoffs, all 18 selected fact IDs, the 36 paired questions, empty result fields, source URLs, hashes, and assertion map. It omits the full copyrighted dataset; reproduce from the licensed source instead of republishing all rows here.

The audit ran as an ordinary-user, dependency-free Python process after the task-scoped Docker route was unavailable. It used no credential, wrote only the receipt, and downloaded no weights. That is weaker isolation than the planned digest-pinned container, so the execution mode is part of the record rather than hidden.

Preserve the source-page discrepancy

The dataset card says the 2,150 facts come from 2,125 Wikipedia pages. The pinned CSV contains 1,883 distinct values in each of page_id, page_title, and item_id: a difference of 242 pages. Multiple facts can come from one page, but that explains why facts exceed pages; it does not reconcile the card’s 2,125 with the observed 1,883.

This is a documentation discrepancy, not evidence that the questions are wrong. It should still block the sentence “we independently verified 2,125 source pages.” The accurate statement is:

Dataset card claim: 2,125 source pages
Pinned CSV observation: 1,883 distinct page IDs
Difference: 242
Status: unresolved documentation mismatch

Keep both values in the receipt. If the publisher revises the card or dataset, create a new revision record instead of altering the old audit. A factuality benchmark loses credibility when its own provenance discrepancies are rounded away.

Freeze a balanced 18-fact pilot

Running 21,500 tasks with eight samples and multiple thinking conditions is an evaluation program, not a smoke test. Start with a pilot that can reveal a broken route, grader, or directional assumption before buying a large run.

Our selection rule is public and deterministic:

Sort all 2,150 popularity values.
Define the bottom quintile as gbc ≤ 914 and the top quintile as gbc ≥ 426,554 for this pinned revision.
For each of the nine named categories, form one low- and one high-popularity candidate set.
Rank each candidate by SHA-256 of the public seed glm52-wikiprofile-v1, category, tier, and fact_id.
Select the first candidate from every set.

That produces 18 unique facts, two per category. The rule is not random at runtime and does not inspect any GLM response. Another engineer with the same CSV obtains the same sample. The one Other row stays in the full dataset but cannot form a balanced low/high category pair, so it is excluded from this pilot and reported explicitly.

The pilot is not statistically representative of every fact. It is a balanced diagnostic. Its low/high split tests the paper’s popularity hypothesis without letting the editor handpick easy examples. If the pilot works, predeclare a larger stratified sample, confidence interval, and stopping rule before examining results.

Pair direct and reverse questions

A direct question usually travels from a familiar subject toward an object. The reverse question asks for the subject from the object-side description. Both address the same underlying proposition, but the access path changes.

One selected pilot record illustrates the structure:

fact_id: 3941070__1
category: Arts
popularity tier: low

direct:
What is the identifier of the seventh DVD volume of Seven of Seven
that included a New Year's special episode?
gold: KIBA-745

reverse:
What Japanese anime television series has KIBA-745 as the identifier
for its seventh DVD volume?
gold: Seven of Seven

The two answers cannot be compared as strings to each other; each has its own gold field. Normalize only formatting differences declared in advance. Do not award credit because the answer appears inside a long speculative paragraph, and do not ask the tested model whether its own response is correct.

For a first route check, the 18 facts yield 36 closed-book questions. Add the natural variants only after the exact pair works. Add completion, contextual, multiple-choice, and thinking conditions when you are ready to classify encoding and recall rather than just directional consistency.

Define a run matrix before calling GLM-5.2

Freeze one row per model, route, condition, and sample. An API alias can move without warning; an open checkpoint can differ from a hosted route. Store the requested model, returned identity when available, endpoint class, date, temperature, seed support, thinking control, output cap, tokenizer or provider revision, and exact prompt hash.

Stage	Facts	Tasks per fact	Samples	Model calls	Decision
Data preflight	2,150	0	0	0	Reject changed or inconsistent input
Paired smoke pilot	18	2	1	36	Verify transport, answer extraction, and directional failures
Repeatability pilot	18	2	8	288	Estimate per-question access probability
Encoding/recall pilot	18	selected completion, context, QA, choice tasks	predeclared	varies	Assign a behavioral profile
Full benchmark	2,150	all 10	predeclared	large	Only after cost, grading, and stopping approval

Do not jump from 36 calls to the paper’s full design because the endpoint accepts the request. Estimate cost from the actual route and include retries, reasoning tokens, grader calls, and human adjudication. Store failures; a timeout is an operational outcome, not a sample to discard silently.

The receipt’s run ledger begins with every answer, correct, and thinking field set to null. A pipeline must change those values only after preserving the raw response, finish reason, usage, and grading evidence. A blank score is more useful than a fabricated baseline.

Grade answers without model self-certification

Exact match is attractive but can reject valid aliases. An unconstrained LLM grader is flexible but can introduce its own bias. Use a staged policy:

Apply Unicode normalization, whitespace folding, and case handling that you specified before the run.
Accept an exact gold or an approved alias from a frozen list.
Send only unresolved cases to an external grader that is not the tested response itself.
Preserve grader model, prompt, raw label, and disagreement reason.
Manually adjudicate a bounded disagreement sample before publishing an aggregate.

The paper generated eight responses at temperature 1 and used separate autoraters. Reproducing its headline method requires those details, not merely the dataset. A one-sample deterministic pilot answers a smaller operational question and must carry a different label.

Test the grader before testing GLM-5.2. Provide one exact answer, one valid alias, one wrong entity, one answer with extra unsupported detail, and one empty/timeout artifact. Confirm that transport failure remains ungraded rather than becoming “model does not know.” Version the grader and regrade saved outputs offline when the rubric changes.

Classify failures before changing the system

Failure signature	What it supports	What it does not prove	Next test
Both direct and reverse wrong	Fact inaccessible in this condition	Permanent absence from parameters	Completion/context and retrieval control
Direct correct, reverse wrong	Directional asymmetry	General hallucination rate	Repeat reverse and natural reverse
No-thinking wrong, thinking correct	Thinking-associated recovery	Guaranteed benefit on other facts	Repeat samples; measure token and latency cost
Multiple choice correct, free answer wrong	Recognition without stable generation	Production answer reliability	Require cited retrieval or constrained choice
Contextual correct, completion wrong	Prompt-format sensitivity	Clean encoding classification by itself	Inspect task construction and repeat
HTTP error or truncation	Route/harness failure	Knowledge failure	Fix transport/cap and rerun under a new receipt

Do not “repair” a failed question by adding the answer to the prompt and then counting it as recall. That becomes contextual extraction. Do not remove rare facts after seeing low scores. That changes the population. Do not combine hosted GLM and self-hosted checkpoint results under one row unless every deployment variable is truly shared.

When retrieval is allowed, label it as a separate system condition. A correct cited answer from a trusted source may be the production winner even if it no longer measures closed-book parametric recall. Product reliability and scientific attribution are different goals; report both honestly.

Turn a pilot into a production gate

Promote only behaviors tied to user risk. For a support assistant, you might freeze twenty product facts and require cited retrieval when closed-book confidence is not enough. For an agent, you might require exact recall of tool permission boundaries and reject action whenever the source is absent. The gate should specify severe errors separately from ordinary misses.

Promote the route only when:
- every run records the exact model and route;
- transport failures are separated from wrong answers;
- severe policy facts have zero unsupported substitutions;
- direct/reverse gaps stay within the declared tolerance;
- thinking and retrieval costs fit the latency and budget caps;
- unresolved answers fail closed or escalate to a trusted source;
- raw outputs and grader revisions can reconstruct the decision.

A pass does not authorize autonomous action. It establishes that a frozen system met a frozen factual contract. Keep tool authorization, prompt-injection defense, data policy, and rollback as separate gates. Our structured-output test covers typed response validation, while the tool-calling guide covers execution boundaries.

For broader score provenance and run-record fields, use the GLM-5.2 benchmark evidence hub. For a small runner that preserves outputs and supports offline rubric changes, the smevals workflow is the natural implementation companion.

Choose a commercial route only after the design gate

The factual test is route-sensitive. A direct API, gateway, and self-hosted checkpoint may expose different revisions, thinking controls, output limits, prices, and observability. Pick a route only after defining the pilot, because the route must preserve the evidence fields the decision needs.

If privacy or revision control requires self-hosting, the local hardware guide explains why the full checkpoint is a multi-accelerator project. Renting hardware is not evidence that the model will recall the facts; the same frozen questions and grader must still run against the deployed artifact.

Audit the sources, method and limits

The topic was discovered through the AI HOT Google Research item. Its title and summary were treated as untrusted discovery hints. Claims were checked against these primary sources:

The fixed Zhipu research discovery page was also checked as a required topic-discovery entry. It did not supply a WikiProfile result and is not used to support one.

The archived audit fetched the dataset CSV and card, verified exact hashes, parsed all rows with Python’s standard library, tested 32 invariants, and generated a deterministic sample. It made no GLM, OpenAI, Gemini, MiniMax, or grader request. The machine’s planned Docker run was abandoned after the ordinary bridge reproduced its known network hang and the one-shot privileged credential route later failed authentication; no repeated privilege attempt was made. All task-named containers were removed.

This page does not claim that WikiProfile measures all factuality, that Wikipedia facts match a production domain, that the benchmark is free of annotation error, or that thinking always improves recall. Its practical contribution is narrower: a pinned, inspectable GLM-5.2 test design with a balanced pilot, an explicit provenance discrepancy, and no invented score.

Answer common factual-recall questions

Did Google test GLM-5.2 on WikiProfile?

No. The paper’s similarly named row is OpenAI GPT-5.2. This audit found no published GLM-5.2 WikiProfile score in the checked primary sources.

Does a wrong answer mean GLM-5.2 never learned the fact?

No. One failure can be missing knowledge, failed recall, phrasing sensitivity, sampling, truncation, route failure, or grading error. Use completion, contextual, direct/reverse, recognition, and thinking controls to narrow it.

Why use direct and reverse questions?

They probe the same proposition from opposite directions. A direct pass and reverse failure reveals access asymmetry that one familiar question would hide.

Is the 18-fact pilot a benchmark score?

No. It is a deterministic smoke sample spanning nine categories and low/high popularity tiers. No model has answered it in this receipt.

Why not run all 21,500 questions now?

The full design needs repeated samples, thinking conditions, grading, cost approval, storage, and failure handling. The 36-question pilot checks the pipeline before that commitment.

Can I use the public JSON as a score?

No. Its answer and correctness fields are intentionally null. It is a data and sampling receipt. A future model-run artifact must use a new revision and preserve raw outputs and external grades.

Does the source-page discrepancy invalidate WikiProfile?

Not by itself. The dataset card reports 2,125 pages; the pinned CSV has 1,883 distinct page identities. The mismatch is unresolved and must be disclosed, but it does not prove the individual question pairs are invalid.

Should factual questions use thinking by default?

Not automatically. Treat thinking as a separate test condition. Measure recovery rate, added tokens, latency, variance, and whether retrieval would be safer or cheaper for the production task.