ZTC-Judge-4B

ZTC-Judge-4B

Answer verification from a single forward pass, with zero generated tokens — at 4B.

ZTC-Judge-4B takes a question and an answer written by any model and scores whether that answer can be trusted. It is the bottom rung of a four-point size ladder measured under one identical protocol, and it is published so the shape of that ladder can be checked rather than asserted.

ZTC — Zero-Token Confidence · Judge — it evaluates someone else's answer, not its own

---

Read this before deploying

This model is not the strongest member of the family, and the card says so with numbers.

A second probe ships with this model, for answers from models we have not seen

The benchmark above is measured on a set where 79% of the answers come from a single model family. A probe fitted on it can end up specialised to that family rather than to correctness. We checked with a model used in neither the fitting nor the scoring — Claude Haiku 4.5 — answering the same 2,018 questions. Those 1,939 answers were graded with the same code and the same domain weights.

ztc_curve_probe_v2.npz is fitted on answers from several models. It clears three gates: it improves on the shipped probe (+0.0342), it clears the surface baseline, and against 8 label-permutation runs it stands at z = 8.41. The file is 10.5 MB, 1,024 anchors, and takes the same call shape as the original — the two are interchangeable in code.

Which file to use. ztc_curve_probe.npz reproduces the benchmark figure above and is the reference for it. ztc_curve_probe_v2.npz is the one to use on answers written by a model outside that benchmark, which is the usual case in deployment.

🔴 The same trade appears at every size. Specialisation helps on the distribution it was fitted to and costs elsewhere. Both files are published so the choice can be made deliberately.

🔑 Across this ladder, 9B and 4B land in the same place under v2 — 0.6931 and 0.6932. Doubling the parameter count buys nothing on this axis. Prefer the 4B unless something else requires the larger model.

Revised 2026-09-21. Earlier revisions of this card reported 0.7364 for Darwin-397B-ZTC and 0.7282 for ZTC-Judge-27B. Those figures were produced by a run whose standardisation statistics were computed over all five domains, including the held-out one, which leaks a small amount of the evaluation domain into every figure. Re-run with the statistics fitted inside the training domains only, the figures are 0.7272 and 0.7289. Cite the current values.

The ladder does not decline smoothly — it steps. Between 9B and 27B the score moves 0.078, while between 4B and 9B it moves 0.015. Whatever carries verification quality is largely absent below 27B on this axis.

Where this model is worth deploying is one specific place, and it is a real one:

🔴 Do not use this model for disaster and safety content. In that domain it does not clear the surface baseline, which means it is reading answer shape rather than correctness there.

✅ Professional-exam style content is where it earns its size. It runs on a laptop, on CPU, and inside networks that never reach the internet — places a hosted API cannot go.

How it works

[question + answer]  →  one forward pass
                     →  final-layer hidden state at the last position (2560-d)
                     →  probe
                     →  score

Generated tokens: 0. No access to the answering model's weights or logits is required; the text of the answer is the only input. Latency is one forward pass, and batching converts directly into throughput.

Usage

import json
import numpy as np, torch
from huggingface_hub import hf_hub_download, snapshot_download
from transformers import AutoModel, AutoTokenizer

REPO = "FINAL-Bench/ZTC-Judge-4B"
cfg  = json.load(open(hf_hub_download(REPO, "ztc_config.json"), encoding="utf-8"))
path = snapshot_download(REPO)

tok   = AutoTokenizer.from_pretrained(path)
model = AutoModel.from_pretrained(path, dtype=torch.bfloat16, low_cpu_mem_usage=True).eval()

def hidden(question, answer):
    b = tok([cfg["template"] % (question, answer)], return_tensors="pt",
            truncation=True, max_length=cfg["max_length"])
    dev = next(model.parameters()).device
    with torch.no_grad():
        h = model(input_ids=b["input_ids"].to(dev),
                  attention_mask=b["attention_mask"].to(dev)).last_hidden_state
    return h[0, int(b["attention_mask"].sum()) - 1].float().numpy().astype(np.float64)

# linear probe — one dot product
p = np.load(hf_hub_download(REPO, "ztc_probe.npz"))
v = hidden("Which defensive chemical does an insect release?", "C. Allomone")
print(float(((v - p["mu"]) / p["sd"]) @ p["w"]))

The score is an unbounded real number; higher means more likely correct. It is a ranking signal, not a calibrated probability — choose a threshold from your own review budget.

Two probes ship with this model

Both read the same input. The curved probe is the one to use when the number matters.

API — drop-in for an existing JEV integration

The endpoint takes the same request shape and returns the same response shape, so switching an existing integration is a URL change.

POST /v1/evaluate
Authorization: Bearer <token>

{"model": "vidraft/ztc",
 "state": {"question": "...", "answer": "..."},
 "questions": {"correct": {"type": "boolean",
                           "instructions": "Is the ANSWER factually correct?"}}}

{"model": "vidraft/ztc-judge-4b",
 "answers": {"correct": {
    "probability": 0.1043,
    "verdict": "review",
    "score": -0.72,
    "position": 0.268,

    "band": "low",
    "action": "hold_or_escalate",
    "measured": {
      "band_accuracy": 0.485,
      "base_accuracy": 0.748,
      "if_lowest_20pct_dropped": 0.814,
      "escalate_gain_at_20pct_budget": 0.0134,
      "do_not": "resample_same_model",
      "why_not": "measured: fixes 6.7% of wrong answers, breaks 13.1% of right ones"}}},
 "usage": {"generated_tokens": 0}}

type accepts boolean and noul. Existing clients read answers.<key>.probability and ignore the rest; the additional fields are there for clients that want to act on the score rather than merely record it. 0.19 s per call, zero generated tokens.

What probability means

The raw score is unbounded. The shipped calibration maps it to P(answer is correct), fitted leave-one-domain-out — the mapping never sees the domain it is applied to.

Measured on the same 2,018 items. ZTC and JEV are effectively tied on calibration; the difference of 0.0016 is not meaningful. Figures published elsewhere for these systems were measured on other test sets and do not reproduce here.

🔴 Calibration is uneven across domains: 0.0225 on biology & medicine, but 0.2941 on scientific reasoning and 0.2381 on general reasoning. Treat the probability as reliable in the first case and as a ranking signal only in the other two.

What to do when the score is low

The score alone is not actionable, so the response carries the measured consequence of each choice.

Three things that work, measured on 2,018 items:

Generate several candidates and let the verifier pick

When the score is low, the most effective next step is not to escalate — it is to produce more candidate answers and select between them. Measured on 346 questions with five candidates each (1,730 candidates), all scored through this endpoint:

The same five candidates swing by 5 points depending on how one is chosen. Majority voting is worse than not resampling at all: when a model prefers a wrong answer, more samples make that wrong consensus more certain. A verifier that ranks the candidates is what turns extra samples into accuracy.

Spend the budget only where it is needed. Generating extra candidates only for low-scoring first attempts captures most of the gain at a fraction of the cost:

At a 30% trigger rate you get three quarters of the benefit for 1.8× the tokens, where always generating costs 4.9× for 1.3× the benefit.

Scope: one generator (GPT-4o-mini), one item set, five candidates. The oracle row shows the headroom that remains — a correct candidate is present far more often than any policy recovers it.

One thing that does not work:

🔴 Do not take a majority vote over resamples. Measured: five resamples with majority voting score 46.53% where a single attempt scores 49.13%. More candidates make a wrong consensus more certain unless something picks between them — see the table above.

Escalation pays for itself through precision, not recall. Re-answering repairs about 38% of wrong answers and damages about 30% of right ones, so a gate is only worth its budget if it mostly calls answers that are actually wrong.

---

Protocol

The same protocol, item set and grading code are applied to every rung of the ladder and to the other systems on the independent leaderboard: https://huggingface.co/spaces/mayafree/typed-decision-leaderboard

Out of scope

• Not a grounding checker. It does not take a source document and decide whether the answer follows from it.
• Not a safety, toxicity or policy classifier.
• Not a calibrated probability. Use it to rank and threshold.
• Not a general-purpose verifier at this size. See the domain table above.

Limitations

• Domain coverage. Scores are meaningful only for the five domains listed. Outside them nothing has been measured and no guarantee is published.
• Below the surface baseline on disaster and safety. Stated in the table rather than omitted.
• Sensitive to which model wrote the answer. The probe is fitted on answers from four models. Adding 1,772 answers from a single additional model shifted the mixture and lowered the size-weighted score from 0.7278 to 0.7177 — professional exams rose to 0.8575 while every other domain fell. Treat "works on any model's output" as a design goal, not a measured guarantee: if your generator differs sharply from the training mixture, measure before relying on the number.
• Revision lock. The probe is fitted to one specific revision of the base model. Running it on a different revision produces no error and silently wrong scores; this repository ships the matching weights so that failure mode cannot occur.
• It reports the verifier's judgement, which is not the answering model's own confidence — that quantity measures 0.5000 on this set.

Lineage

What this repository contains

The rest of the ladder

ZTC-Judge-27B · Darwin-397B-ZTC · ZTC-Judge-9B · ZTC-Judge-4B

License

The base model is Apache-2.0 and redistributable. The probes, the inference code and the evaluation tables are assets of FINAL-Bench / VIDRAFT.