oev-typed

<p align="center"> <img src="https://raw.githubusercontent.com/divyanshudhruv/oev/refs/heads/main/assets/banner_typed.png" alt="OEV-typed banner" width="100%" /> </p>

A 184M-parameter decision engine. State + typed questions in, calibrated probability distributions out, one forward pass, 22.2 ms p50 on T4.

OEV reads a state, scores typed questions over it in one forward pass, and returns calibrated probability distributions. There is no text generation step, so there is nothing to parse and nothing to hallucinate.

It answers three kinds of questions over any text state: choice (pick a label), noul (yes/no), and score (ordinal levels). All options are packed into one sequence as anchors, and a single head scores them all at once.

Results (typed-decisions benchmark, 2,000 decisions)

Fine-tuned on the benchmark's train split, same protocol as laya-typed-decisions. OEV numbers were measured on a single Tesla T4; Jev and laya values are quoted from their published tables.

The temperature-calibrated row is the best calibrated single: temperature 0.598 fitted on the valid split only, accuracy unchanged by monotone scaling. Use it via OEV(checkpoint, temperature=0.598). | OEV ensemble (all four checkpoints, equal votes) | 4 × 184M | 0.7760 | 0.5830 | - | - |

Per workflow (ensemble): invoice 0.836, customer service 0.804, agent-trace 0.740, security incidents 0.722 (OEV leads three of the four workflows). Per primitive: noul 0.853, choice 0.748, score 0.738.

Checkpoints

For the ensemble results, average the softmax probabilities of the members with equal weights: typed-decisions 0.7760 (four files), Banking77 0.8529 with ECE 0.0595 (three files: b77-oev-tiny.pt + b77a-oev-tiny.pt + b77b-oev-tiny.pt).

Usage

import torch, torch.nn.functional as F
from oev.evaluate import load_model
from oev.tokenizer_hf import HFTokenPacker
from huggingface_hub import hf_hub_download

path = hf_hub_download("divyanshudhruv/oev-typed", "student-r3-oev-tiny.pt")
model = load_model(path, "cuda")
packer = HFTokenPacker(model.cfg["backbone"])

state = "We were charged twice for the same order."
question = {"name": "department", "type": "choice",
            "instructions": "Which department should handle this?",
            "options": ["billing", "technical", "other"], "answer": "billing"}

# The Jev/TypeSafe `criteria` schema (used by laya and Kev) is also accepted:
# {"name": "department", "type": "choice", "instructions": "...",
#  "criteria": {"billing": "invoices, payments", "technical": "bugs, outages"}}
# descriptions are ignored by OEV; the option names are what it scores.

ids, anchors, _ = packer.pack(state, question, model.cfg["max_len"])
with torch.no_grad():
    logits = model(torch.tensor([ids]).cuda(),
                   torch.zeros(1, len(ids), dtype=torch.bool).cuda(),
                   torch.tensor([anchors]).cuda())
probs = F.softmax(logits[0].float(), dim=-1)
# probs ≈ billing 0.94, technical 0.04, other 0.02

Try the interactive Space demo, or install the package:

pip install oev

Training and evaluation code, the ensemble command, dataset converters and tests are all in the GitHub repo.

Training recipe

1. Soft-target fine-tune - typed-decisions train split, teacher's full distributions as targets, full-state packing, DeBERTa-v3-base backbone, differential LRs (head 2e-4 / backbone 2e-5), 4 epochs, batch 8, 768 tokens.
2. Staged variants - multi-task pretrain (AG News + emotion + typed), then typed polish, then weight soup.
3. RLCD - REINFORCE with a Brier-reward against the teacher's stored probabilities, batch-mean centered, blended with the imitation anchor (α = 0.5). Two seeds + one soup-init variant.
4. Ensemble - equal-weight probability averaging across the four checkpoints.

The whole pipeline trains in about a day on one T4. train_colab.ipynb runs it end to end.

Notes

• Fine-tuned on each benchmark's own train split; comparison numbers are from the respective published tables.
• OEV trains to match the teacher's full distributions (training Brier vs teacher: 0.058).
• English models. OEV latency is p50 of 50 warmed runs including sync on T4.

License

Apache 2.0. The interface and benchmark protocol follow Laya and the decision-model category introduced by TypeSafe's Jev; their published numbers are quoted for comparison and remain their measurements.

Banking77

Fine-tuned checkpoints: b77-oev-tiny.pt plus two warm-started re-tunes (b77a-oev-tiny.pt, b77b-oev-tiny.pt). Each of the 77 intents is embedded as its own anchor occupying the full token budget, so no label is truncated.

[!WARNING] The published Jev Banking77 figure uses 72 labels, while OEV uses 77. The values are not a controlled head-to-head comparison.

The soup (weight-space average of the three members) beats the probability ensemble with a single 735MB artifact and reaches coverage 0.7604 at a <=5% error budget (AURC 0.0389), so 76% of decisions can be automated under the stated budget. On CPU the ONNX INT8 build runs at 54.2 ms p50 on 8 threads (228 MB artifact); the release measurement with torch fp32 was 447 ms. No competing decision model publishes any CPU latency.