veda-v11

VEDA-v11 — Parallel System-One Decision Model (149M)

An open, Jev-shaped typed-decision model. One text state plus any number of typed questions (choice / score / noul) goes in; a calibrated probability distribution per question comes back from one forward pass. Nothing is generated, so the structured-output error rate is 0 by construction.

• Backbone: answerdotai/ModernBERT-base (149,754,243 params, fully fine-tuned)
• Architecture: shared state encoding (encode once, O(S+slots)) + [MASK] marker pooling + 2-layer decision head + learned per-decision temperature + abstain head (wired, untrained)
• Loss: soft cross-entropy + Brier + RPS (ordinal) + KL trust-region
• Training: 136,398 rows (EAGLE 77,898 + TITAN 58,180 + GEM 320), 2 epochs on 2×T4
• License: MIT (model code/weights). Training data: EAGLE CC0, TITAN CC-BY-4.0 (see attribution below)

Measured results (all raw, single seed)

Reading these honestly: the model learned its training distribution (EAGLE/TITAN phrasing) well, including good in-family calibration — but does not generalize to external preference judgments or novel templates, where it falls back to a first-option prior. See Limitations.

Known limitations (measured, not suspected)

1. Position bias. 55.3% of training golds sit at option position 0. Reversing option order on in-distribution cal drops accuracy 79% → 55% (n=58). Prescribed fix: shuffle criteria at mixture-build time (specified, not yet trained).
2. No vision / grounding. Text states only. Not a computer-use model on its own; it is the decision layer such an agent would call. TIGER-S1 grounding integration is future work.
3. English-first. ModernBERT-base backbone; multilingual performance unmeasured, expect weak.
4. 77+ option sets exceed the 64-marker training budget (Banking77 run with eval-side 1024-token window; results pending).
5. Tool selection only — picks tool names, does not generate arguments or multi-turn chains.

Use

import torch
from transformers import AutoTokenizer, AutoModel
# backbone + head weights in veda-v11.pt (state_dict of ParallelDecisionModel;
# see https://github.com/Sravanjangam/veda/blob/main/veda/model.py)

Packing format (state once + [MASK] per option) is implemented in veda/pack.py in the repo above.

Training data attribution

• EAGLE (Open-Jev, Zefan-Cai): CC0 — 79,116 train rows used
• TITAN (xLAM-60k): CC-BY-4.0 — 60,000 tool-calling rows used
• GEM: 320 hand-crafted seeds (own)
• Calibration split deduplicated against train (zero exact state-question overlap)

Citation

Comparison references: JevBench (jevbench.xyz), AY Automate Jev-vs-LLM benchmark (Sep 2026), Aman Kumar "Jev measured" (Sep 2026), JEV-as-a-Judge (arXiv 2609.26550). Laya reference numbers (typed-decisions 0.766, Banking77 0.425, raw ECE 0.213) per prior research notes.