Model Card for ThatsNinja/glm-edge-falsifier-1.5b
A falsification-gated abstention model built on a frozen GLM-Edge 1.5B Chat
base. The model generates multiple counterfactual narratives, a calibrated
verifier discards those unlikely to hold, and it commits only to survivors —
abstaining ("I don't know") when none remain. Block-sparse attention and
latent KV compression target small-hardware efficiency.
Model Details
Model Description
GLM-Edge-Falsifier-1.5B is not a fine-tune of the base weights. The base
GLM-Edge 1.5B Chat is kept frozen; the model is an inference-time
architecture wrapped around it, consisting of:
- Falsification fast loop (RFC0013) — at inference, $N$ candidate
narratives are sampled (temperature 0.8, top-$p$ 0.95). Each is scored by
two channels: a consistency channel (semantic clustering across samples)
and a verifier channel (a calibrated MLP over on-policy attention
features). A Popperian max-combination yields a combined falsification
score $\phi(c)$. The surviving set is $S = {c : \phi(c) < \tau}$. If
$S \neq \emptyset$, the model commits to $\arg\max_{c \in S} \hat{u}(c)$;
otherwise it abstains ("I don't know") rather than guessing.
- Correctness verifier (RFC0002/RFC0012) — a small MLP trained on
on-policy features extracted from the base model's own generations, with
truth-anchored gold-answer-match labels (GSM8K). Calibrated with
temperature scaling (Guo et al. 2017).
- Learned deactivation slow loop (RFC0013) — per-region Beta posteriors
on adverse-outcome rates drive a block-sparse attention mask, deactivating
attention regions once the posterior is confident and well-observed
($n \ge n_{\min}$). Implemented and self-test passing, but not yet wired
to the HIP kernel for an end-to-end generation run — no generation results
in this card use learned deactivation.
- Block-sparse SDPA kernel (RFC0001/0003) — a custom HIP kernel
(BLOCK_M=BLOCK_N=32, HEAD_DIM=128) implementing online-softmax block-sparse
attention, validated against a CPU reference (max error 5.4e-7 at S=64/128).
A monkey-patch hooks it into GLM-Edge's attention for experimentation, but
no results in this card use the custom kernel — all generation results use
PyTorch native SDPA. GQA-aware (16 Q heads, 4 KV heads). BF16 integration is
in progress.
- Latent KV compression (RFC0005, MLA-style) — down/up-projection
adapters ($W_{kv_a}$, $W_{kv_b}$, RMSNorm) compress KV cache to a rank-512
latent. Projections trained, quality target met (eval cosine 0.9547,
2× KV compression).
- FP8 block quantization (RFC0006) — per-128-block e4m3 quantization for
weights and KV cache. Path A (Python dequant) validated; in-kernel
dequant (Path B) is future work.
- Process reward model (RFC0014/RFC0017) — a GRU-based step-level
verifier (~15K params) trained on Math-Shepherd-style Monte-Carlo step
labels (1000 problems, 7064 steps), replacing the token-level ORM
verifier for hierarchical success measurement. Step-level ECE 0.0269 —
breaks the 0.05 floor at the level N* stopping and tree search operate
at. Includes a task head trained on solution_correct (task-level ECE
0.1048, roughly at parity with the ORM's 0.0910 task-level baseline).
Note (2026-08-10): the 30-dim enriched pooling variant (mean+max+std,
30 features) was tested and produced worse calibration (step ECE 0.0339,
task head ECE 0.1498) despite +1.4pp step accuracy — evidence that the
precision-cost curve is U-shaped and we're past the optimal measurement
spend $m^*$ at the step level. See the measurement-cost model
(docs/theory/measurement_cost_model.md).
- Conformal trajectory calibration (RFC0016) — split conformal over PRM
step scores delivers distribution-free coverage guarantees: per-step
coverage 0.9622 and trajectory coverage 0.9452 (target 0.90, α=0.1).
- Falsification-guided tree search (RFC0015) — PRM-guided beam search
with backtracking. Beats flat best-of-N at matched compute on hard
domains: 61.7% vs 47.9% on GSM8K (n=100, +13.8pp). Ties on the easy
domain SVAMP (71.4% vs 71.4%) — search earns its compute where the base
model is weak (per Qin et al. 2025).
- Live step-gated generation (RFC0018 draft) — TRUE live inference-time
gating (
steer_generate.py): KV-cached incremental decode; the PRM +
conformal threshold scores each completed step and stops decoding
immediately on a fired gate, so tokens after the gate never compute.
Dev slice (n=10): the tau 0.45 gate saves 31% of tokens at 0pp accuracy
loss; the conformal gate adds +10pp accuracy while saving 9.2% of
tokens. Supersedes the post-hoc N* stopping mode, which computed all
tokens and only truncated afterwards.
- Developed by: Jonah Sullivan (ThatsNinja)
- Funded by [optional]: Self-funded
- Shared by [optional]: ThatsNinja
- Model type: Decoder-only transformer with inference-time falsification gating and abstention
- Language(s) (NLP): English
- License: MIT for the falsifier architecture and code. The base GLM-Edge
1.5B Chat weights retain the GLM-4 model license.
- Finetuned from model [optional]: Not fine-tuned — wraps a frozen
zai-org/glm-edge-1.5b-chat base. Only small adapter layers (MLA
projections, verifier MLP) are trained.
Model Sources [optional]
- Repository: TheRock / correctness_model
- RFCs:
correctness_model/docs/rfcs/ (RFC0001–RFC0018)
- Results:
correctness_model/docs/RESULTS.md
- Demo [optional]: N/A (research prototype)
Uses
Direct Use
Research on falsification-based generation, calibrated abstention, and
verifier-guided decoding. The model is not a general-purpose chatbot — it
is optimized for correctness over fluency, trading coverage for
precision. On the full GSM8K test set (n=1319) it commits to 53.8% of
problems and abstains on 46.2%, achieving 68.6% accuracy on what it commits
to (vs 36.0% greedy baseline).
Downstream Use [optional]
- As a verifier-augmented inference harness for other frozen base models
(the architecture is base-model-agnostic in principle; in practice the
feature extractor hooks are written for GLM-Edge's layer structure and would
need adaptation for other model families).
- As a calibrated abstention layer for high-stakes QA where "I don't
know" is preferable to a confident wrong answer.
Out-of-Scope Use
- General open-ended chat (the model is tuned for single-answer tasks with
checkable correctness, not conversation).
- Tasks without a notion of answer correctness (creative writing,
summarization quality is unevaluated).
- Safety-critical decisions without a human in the loop — abstention is
structural, not a safety guarantee.
- Domains beyond grade-school math are only partially verified: SVAMP
transfer is measured (committed 77.2%), but broader domains are not.
Bias, Risks, and Limitations
- Verifier overfitting: Trained on 1000 GSM8K problems with greedy
generation only. The token-level ECE floor is ~0.06 (best sweep config
0.0597); the gap was structural at the token level and was addressed by
hierarchical measurement (RFC0017) at the step level, not by tuning. The
verifier likely overfits to GSM8K arithmetic patterns.
- Abstention is learned, not principled: The 46.2% abstention rate is
an emergent property of $\tau=0.6$ and the verifier's calibration, not a
guarantee of epistemic uncertainty. Conformal calibration (RFC0016)
provides a distribution-free coverage guarantee for step/trajectory
commit decisions; the raw abstention rate does not.
- Kernel is fp32, model is BF16: The HIP kernel is validated in fp32;
BF16 integration is in progress. Kernel-integrated results use PyTorch
native SDPA, not the custom kernel.
- Attention features require eager attention: GLM-Edge's default SDPA
attention implementation does not expose attention weights
(
output_attentions=True returns an empty tuple). Feature extraction
loads the base with attn_implementation='eager'.
- PRM task-level parity: The step-level verifier (RFC0014) is trained
on 1000 problems and reaches step-level ECE 0.0269 (< 0.05). At the task
level its ECE (0.1048) is at rough parity with the ORM baseline (0.0910),
so the ORM verifier remains the production task-level channel for now.
- N* stopping result reclassified (2026-09-17): the earlier ~10% token
reduction at ≤2pp accuracy loss was post-hoc accounting —
generate_with_step_stopping() decoded all tokens and only truncated
afterwards, so no compute was saved. The live gate (RFC0018,
steer_generate.py) is the corrected mechanism: dev-slice (n=10) it saves
31% of tokens at 0pp accuracy loss (tau 0.45). Step discrimination (step
accuracy 0.70) remains the binding constraint on how aggressive the gate
can be.
- Measurement-cost model is a hypothesis: the claim that the
precision-cost curve $\varepsilon(m)$ is U-shaped with a finite optimal
measurement spend $m^*$ rests on one supporting data point (the 30-dim
enriched-pooling negative result). It is a research direction, not an
established result.
- Single-dataset evaluation: All metrics are on GSM8K (plus SVAMP
transfer). MATH/MMLU adapters exist but no cross-domain training or
evaluation on them has been conducted yet.
- Base model biases inherited: As a frozen-base wrapper, all biases,
hallucination modes, and limitations of GLM-Edge 1.5B Chat are inherited
unmodified.
Recommendations
Users (both direct and downstream) should be made aware that:
- The model abstains by design — empty output or "I don't know" is a
first-class outcome, not an error.
- Committed answers are substantially more likely to be correct than
greedy decoding (68.6% vs 36.0% on the full GSM8K test set), but this is
measured only on GSM8K.
- The verifier should be retrained on the target domain before use
outside grade-school math.
- For calibrated uncertainty, use the conformal channel (RFC0016), not
the raw abstention rate; per-step coverage is 0.9622 at α=0.1 by
construction.
How to Get Started with the Model
# 1. Install (ROCm / AMD Instinct) — use the STABLE rocm6.4 index.
# The nightly multi-arch index installs a wheel that reports gfx942 in
# get_arch_list() but fails every GPU op with hipErrorInvalidImage.
pip install torch torchvision torchaudio \
--index-url https://download.pytorch.org/whl/rocm6.4
pip install -r correctness_model/requirements-rocm.txt
# 2. Download the frozen base model
python -c "from huggingface_hub import snapshot_download; \
snapshot_download('zai-org/glm-edge-1.5b-chat', \
local_dir='correctness_model/models/glm-edge-1.5b-chat')"
# 3. Run falsification-based generation with the trained verifier
python correctness_model/src/python/falsify_generate.py \
--gsm8k --num-problems 50 --num-samples 6 \
--tau 0.6 --verifier correctness_model/checkpoints/verifier-v2-1k.pt
For the full training pipeline (MLA projections, verifier, feature
collection) and remote AMD Instinct setup, see
correctness_model/scripts/setup_remote.sh.
Training Details
Training Data
Verifier (RFC0002/RFC0012):
- Dataset: GSM8K train split (Cobbe et al. 2021), 1000 problems
- Labels: Truth-anchored gold-answer match (Track 1, RFC0012). The
model generates greedily, the predicted answer is extracted and compared
to the gold
#### <number> answer. Label is outcome-based (ORM-style),
broadcast across the generated sequence.
- Features: On-policy — extracted from the model's own generations via
attention hooks (per-head attention entropy, output stats, cross-head
consistency), closing the teacher-forcing distribution gap. Requires
attn_implementation='eager' (SDPA does not expose attention weights).
- Cached as: Parquet in
correctness_model/data/gsm8k_features_1k/
Process reward model / step verifier (RFC0014/RFC0017):
- Dataset: GSM8K, step-level labels via Math-Shepherd-style Monte-Carlo
Hard Estimation (N=8 completions per step). 1000 problems collected
(7064 steps, 63.2% positive).
- Features: Per-step mean-pooled token features + step metadata
(12-dim step embedding), extracted with
attn_implementation='eager'.
- Cached as: Parquet in
data/step_labels/ (labels + features).
MLA projections (RFC0005):
- Dataset: GSM8K train prompts (in-distribution)
- Objective: Reconstruction MSE — compress each layer's projected K/V
to a rank-$r$ latent and reconstruct; minimize $|K - \hat{K}|^2 + |V - \hat{V}|^2$.
Base model frozen; only $W_{kv_a}$, $W_{kv_b}$, RMSNorm trained.
- Status: Trained at ranks 64/128/256/512. Rank 512 meets the quality
target (eval cosine 0.9547 > 0.95, 2× KV compression).
Training Procedure
Preprocessing [optional]
- GSM8K gold answers extracted via
#### <number> regex, normalized (strip
commas, $, %, canonicalize numerics).
- Prompts built with the GLM-Edge chat template (
build_prompt).
- Features normalized with
FeatureNormalizer (z-score) fitted on the
training split only.
Training Hyperparameters
Verifier MLP:
- Architecture: MLP, hidden_dim 256 (best sweep config), 9 input features:
attention_entropy, attention_max_weight, output_variance, output_saturation,
output_magnitude, cross_head_consistency, hidden_norm, token_entropy,
token_confidence
- Loss: Binary cross-entropy
- Optimizer: Adam, lr 3e-3
- Epochs: 50
- Split: Sequence-level 70/15/15 (no token leakage between splits)
- Calibration: Temperature scaling (Guo et al. 2017), fitted on
validation logits, evaluated on held-out test
- Training regime: BF16 mixed precision (base model), fp32 (verifier MLP)
Step verifier / PRM (RFC0014):
- Architecture: GRU, input_dim 12, hidden_dim 256, 2 layers (~15K params)
- Loss: Binary cross-entropy over per-step logits (masked padding)
- Optimizer: Adam, lr 1e-3
- Epochs: 200
- Split: Problem-level 70/15/15 (no step leakage between splits)
- Task composition: a task head (Linear on GRU final state) trained directly
on
solution_correct outperforms both geometric-mean and min composition of
step probabilities for task-level scoring (task head ECE 0.1048 vs geomean
0.1284 vs min 0.1203 on 1000 problems). The geomean + temperature scaling
variant was the best composition-based approach in the 200-problem pilot but
is superseded by the task head at scale.
MLA projections:
- Architecture: Per-layer $W_{kv_a}: 1024 \to 512$, RMSNorm, $W_{kv_b}: 512 \to 1024$
- Loss: Reconstruction MSE (K + V)
- Optimizer: AdamW, lr 1e-3
- Init: $W_{kv_b}$ small-normal (std 0.02) for near-zero initial reconstruction
- Training regime: BF16 base (frozen), fp32 loss accumulation
Speeds, Sizes, Times [optional]
| Component | Size | Notes |
|---|
| Base model (frozen) | 2.945 GB (BF16) | GLM-Edge 1.5B, 1.472B params |
| Verifier MLP | ~0.3 MB | 9-dim input, hidden 256 (best sweep config) |
| Step verifier (PRM) | ~60 KB | GRU, 12-dim input, hidden 256, 2 layers |
| MLA projections (28 layers) | ~27 MB | rank-512, per-layer adapters (quality target met) |
| KV cache (full, per 1K tokens) | ~0.054 GB | 28 layers × 4 KV heads × 128 head_dim × 2 bytes (BF16) × 2 (K+V) |
| KV cache (MLA rank-512, per 1K tokens) | ~0.027 GB | 2× reduction vs full KV (stores rank-512 latent instead of separate K+V; validated, cosine 0.9547) |
| FP8 weight quant | 1.45 GB | 2× reduction vs BF16 weights (validated Path A) |
Evaluation
Testing Data, Factors & Metrics
Testing Data
- GSM8K test split, full set (n=1319) for falsification evaluation;
150 sequences (39,690 tokens) held out for verifier test metrics.
- SVAMP (ChilleD/SVAMP, n=300) for cross-domain transfer.
- No external benchmarks (MMLU, TruthfulQA) evaluated yet.
Factors
- Commit vs abstain: The primary factor is whether the model commits or
abstains, driven by the threshold $\tau$ and the surviving-set cardinality.
- Number of samples $N$: 6 narratives per problem (temperature 0.8,
top-$p$ 0.95).
- Threshold $\tau$: 0.6 (the falsification cutoff).
Metrics
- Committed accuracy: accuracy over problems the model commits to (the
precision-oriented metric this model optimizes for).
- Abstention rate: fraction of problems where the surviving set is empty.
- Greedy baseline accuracy: single greedy decode accuracy (control).
- Verifier F1 / ECE: held-out token-level F1 and expected calibration
error for the verifier channel.
Results
Verifier (1000-problem, held-out test, 150 sequences, 39,690 tokens)
| Metric | Value |
|---|
| Accuracy | 0.7171 |
| Precision | 0.8021 |
| Recall | 0.4837 |
| F1 | 0.6035 |
| ECE | 0.0893 |
| Fitted temperature | 0.990 |
Scaling training data from 200 to 1000 problems halved ECE (0.18 → 0.089)
and raised precision to 0.80. F1 of 0.60 on a 41%-positive task is well
above the 0.41 chance baseline. A subsequent hyperparameter sweep improved
ECE further to 0.0597 (best config: hidden_dim 256, lr 3e-3, 50 epochs).
Falsification-based generation (full GSM8K test set, n=1319, $N=6$, $\tau=0.6$)
| Metric | Value |
|---|
| Committed | 710 (53.8%) |
| Abstained | 609 (46.2%) |
| Committed accuracy | 68.6% |
| Greedy baseline | 36.0% |
Committed accuracy beats the greedy baseline by 32 percentage points on the
full test set. These results use the token-level ORM verifier
(verifier-v2-1k.pt, ECE 0.0893) as the verifier channel — not the
step-level PRM (which was trained later and has not been wired into
falsify_generate.py). The consistency channel uses semantic clustering
across the 6 samples. The combined falsification score is
$\phi(c) = \max(\phi_{\text{verifier}}, \phi_{\text{consistency}})$; a
candidate survives if $\phi(c) < 0.6$.
Component validation
| Component | Metric | Result | Status |
|---|
| Block-sparse SDPA kernel | Max error vs CPU ref (S=64/128) | 5.4e-7 | ✅ validated |
| FP8 weight quant (Path A) | Cosine sim vs BF16 | 0.9990 | ✅ math validated |
| FP8 weight quant (Path A) | Weight rel-err | 2.25% | ✅ |
| FP8 weight quant (Path A) | Same top-1 prediction | yes | ✅ |
| FP8 KV cache (Path A) | Round-trip cosine sim | 0.9996 | ✅ |
| MLA projections (rank 512) | Eval cosine sim vs full attn | 0.9547 | ✅ quality target met (>0.95) |
Process reward model / step verifier (RFC0014/RFC0017)
Trained on 1000 problems of Math-Shepherd-style Monte-Carlo step labels
(7064 steps, 63.2% positive). Task label is solution_correct (balanced) —
the fair ORM-comparable label.
| Metric | Value |
|---|
| Step-level accuracy | 0.7005 |
| Step-level ECE | 0.0269 ✅ (breaks the 0.05 floor) |
| Task-level ECE (task head) | 0.1048 |
| Task head accuracy | 0.7365 |
| ORM task-level baseline (mean-of-tokens) | 0.0910 |
RFC0017 is validated at the step level (ECE 0.0269 < 0.05) — the level
that N* stopping and tree search operate at. At the task level the PRM is
at rough parity with the ORM (0.1048 vs 0.0910); closing that gap is open
work. Note the earlier "0.0597 ORM floor" was a token-level metric, not
comparable to task-level ECE.
Summary
Falsification gating lifts committed accuracy from 36.0% to 68.6% on the
full GSM8K test set (n=1319, $N=6$, $\tau=0.6$) with structural abstention,
and transfers cross-domain to SVAMP (committed 77.2%, n=300). These results
use the token-level ORM verifier (verifier-v2-1k.pt); the step-level PRM
has not yet been wired into the falsification loop. Scaling the verifier from
200 to 1000 problems plus a hyperparameter sweep brought token-level ECE from
0.18 to 0.0597, and MLA projections meet the quality target at rank 512
(eval cosine 0.9547).
The token-level ECE has hit a structural floor at ~0.06: calibration-side
levers (hyperparameters, richer calibrators) are exhausted. The remaining gap
is on the feature/supervision side. RFC0017 (Hierarchical Success Measurement)
addresses this by dividing success measurement across token/step/task levels;
the step-level PRM reaches step-level ECE 0.0269 (below 0.05) but task-level
ECE 0.1048 (at parity with the ORM baseline 0.0910, not yet better). The
exquisite-token stopping rule ($N^$, RFC0010) is not yet implemented in
live generation — it is blocked on trustworthy calibration, and the
step-level N threshold sweep did not meet the combined target (≥20% token
reduction AND ≤2pp accuracy loss; best result ~10% reduction at ≤2pp loss).
Remaining hardware work: kernel integration into the model (BF16) and FP8
Path B (in-kernel dequant).
Model Examination [optional]
- Per-head attention entropy and cross-head consistency are among
the 9 verifier features; their distributions differ between correct and
incorrect generations (the basis for the F1 0.60 signal on the 1000-problem
verifier, or 0.65 for the best sweep config). Note: these features require
attn_implementation='eager' — GLM-Edge's default SDPA implementation
returns an empty attention tuple.
- Temperature scaling fits a temperature near identity (0.990), and
richer calibrators (vector, histogram) overfit — the ~0.06 ECE floor is
structural (feature/supervision side), motivating RFC0017's hierarchical
success measurement.
- Step-level measurement (RFC0017): the GRU step verifier trained on
1000 problems reaches step-level ECE 0.0269 — below the 0.05 floor,
confirming that step-level labels + sequential context break the floor at
the step level. Its task head (trained on solution_correct) reaches
task-level ECE 0.1048, at rough parity with the ORM's 0.0910.
- Learned deactivation builds Beta posteriors per attention region; the
self-test passes but the mask has not been wired to the HIP kernel for an
end-to-end generation run.
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: AMD Radeon RX 6900 XT (gfx1030, 16GB GDDR6, TDP 300W) for development; AMD Instinct MI300X (gfx942, 192GB HBM) for remote training.
- Hours used: Development: ~20 hrs local GPU (estimate). Remote (MI300X):
~12 hrs (estimate; MLA rank sweep, verifier scaling + sweep, falsification
evals, step-label collection). Step-label collection alone was ~5.1 hrs for
1000 problems (306.5 min); total remote time is an estimate and may be
higher.
- Cloud Provider: DigitalOcean (AMD Instinct GPU droplets).
- Compute Region: NYC2.
- Carbon Emitted: [More Information Needed] — the base model is frozen
and only small adapters are trained, so the marginal training carbon is
minimal compared to a full fine-tune. The dominant cost is Monte-Carlo
step-label generation (inference, not training). Hour estimates are
approximate and should be treated as order-of-magnitude.
Technical Specifications [optional]
Model Architecture and Objective
Base: GLM-Edge 1.5B Chat — 28 layers, hidden_size 2048, 16 query heads,
4 KV heads (GQA 4:1), head_dim 128, BF16, RoPE positional encoding.
Falsifier wrapper (the contribution):
$$
S = {c_i : \phi(c_i) < \tau}, \quad
\phi(c) = \max\big(\phi_{\text{verifier}}(c),\ \phi_{\text{consistency}}(c)\big)
$$
$$
\text{output} = \begin{cases}
\arg\max_{c \in S} \hat{u}(c) & \text{if } S \neq \emptyset \
\text{"I don't know"} & \text{if } S = \emptyset
\end{cases}
$$
The exquisite-token stopping rule (RFC0010) sets $N^$ where marginal
token utility equals average utility — past $N^$, each token destroys
value. This is the economic framing; mid-sequence early stopping at $N^*$
is future work (blocked on ECE < 0.05).
Compute Infrastructure
Hardware
- Development: AMD RX 6900 XT (gfx1030, RDNA2, 16GB GDDR6)
- Training: AMD Instinct MI300X (gfx942, CDNA3, 192GB HBM3) via
DigitalOcean GPU Droplets
Software
- PyTorch 2.9.1+rocm6.4 (MI300X) / 2.12.0+rocm7.15 (gfx1030 local)
- transformers 5.15.0.dev0 (from source)
- accelerate 1.14.0, datasets 5.0.1
- HIP/ROCm (block-sparse kernel), CMake + Ninja build
- Python 3.11+
Citation
This model
@misc{glm-edge-falsifier-1.5b,
author = {Jonah Sullivan},
title = {GLM-Edge-Falsifier-1.5B: Falsification-Gated Abstention},
year = {2026},
note = {Research prototype},
url = {https://huggingface.co/ThatsNinja/glm-edge-falsifier-1.5b}
}
Prior work this build depends on
- Cobbe et al. 2021 — Training Verifiers to Solve Math Word Problems
(arXiv:2110.14168). GSM8K; ORM-style
outcome-supervised verification.
- Wang et al. 2023 — Math-Shepherd: Verify and Reinforce LLMs Step-by-step
without Human Annotations (arXiv:2312.08935).
Monte-Carlo Hard Estimation step labels (our H2 labeler).
- Lightman et al. 2023 — Let's Verify Step by Step
(arXiv:2305.20050). Process supervision
at the step level (PRM800K); motivates step-level credit assignment.
- Uesato et al. 2022 — Solving math word problems with process- and
outcome-based feedback (arXiv:2211.14275).
Early PRM vs ORM comparison.
- Guo et al. 2017 — On Calibration of Modern Neural Networks
(arXiv:1706.04599). Temperature scaling.
- Gibbs & Candès 2021 — Adaptive Conformal Inference Under Distribution
Shift (arXiv:2106.00170). Online
conformal calibration (RFC0016 Layer 3).
- Angelopoulos et al. 2024 — Conformal Risk Control
(arXiv:2209.07411) and related conformal
prediction literature. Coverage-guarantee framework.
- Qin et al. 2025 — Tree Search for Language Model Agents (backtracking
is not universally beneficial). Per-task justification for RFC0015 tree
search; flat best-of-N as mandatory control.
- DeepSeek-AI 2024 — DeepSeek-V2 (arXiv:2405.04434)
and DeepSeek-V3 (arXiv:2412.19437).
Multi-head Latent Attention (MLA), basis for RFC0005 latent KV compression.
- Zelikman et al. 2022 — STaR: Bootstrapping Reasoning With Reasoning
(arXiv:2203.14465). Self-improvement via
generated rationales; related to our on-policy self-labeled collection.
- LeCun 2022 — A Path Towards Autonomous Machine Intelligence
(JEPA / world-model architecture). Evaluated as a potential complement
(latent-prediction scoring channel); not used in this build.
Base model
GLM-Edge 1.5B Chat (zai-org/glm-edge-1.5b-chat) — frozen base, GLM-4 model
license. Not modified by this work.
Glossary [optional]
- Falsification (Popperian): A narrative is discarded if any channel
marks it as unlikely to hold (max-combination), mirroring Popper's
asymmetry — you can refute but not confirm.
- Surviving set $S$: The candidates that pass falsification at threshold
$\tau$; the model commits to the highest-utility member.
- Abstention: Structural refusal to answer when $S = \emptyset$; a
first-class output, not an error.
- Exquisite tokens (RFC0010): Tokens worth emitting — the optimal
stopping point $N^*$ where marginal utility equals average utility.
- MLA (Multi-head Latent Attention): Compresses per-head K/V into a
low-rank latent via down/up-projections, reducing KV cache memory
(DeepSeek-V2/V3 architecture).
- FP8 (e4m3): 8-bit float format (1 sign, 4 exponent, 3 mantissa bits;
max 448.0), used with per-128-block scaling for weight/KV quantization.
- GQA: Grouped-Query Attention — 16 query heads share 4 KV heads (4:1),
already reducing KV cache vs MHA.
- ECE: Expected Calibration Error — the gap between predicted confidence
and empirical accuracy; target < 0.05 for the $N^*$ stopping rule.
More Information [optional]
Architecture decision records: correctness_model/docs/rfcs/RFC0001–RFC0017.
Scaling roadmap: correctness_model/docs/SCALING_PLAN.md.
Reproducible results: correctness_model/docs/RESULTS.md.
Theory: correctness_model/docs/theory/ (measurement-cost model, training pipeline design).
Model Card Authors [optional]
Jonah Sullivan (ThatsNinja)
Model Card Contact
Jonah Sullivan — [email protected]