ouro-2.6b-decision-lora

ouro-2.6b-decision-lora

A calibrated decision model: state in, typed probability distributions out, one forward pass, no text generation. Three question types, the same shape as TypeSafe's Jev API: choice (distribution over options), score (distribution over ordered levels, plus the expected level) and noul (P(true)).

It is a LoRA adapter (r=16, 1.2 % of parameters) plus a 2-layer scoring head on top of ByteDance/Ouro-2.6B, a looped language model (48 layers, 4 recurrent steps). The head reads the last-token hidden state of state + question + listed options + "Answer: <letter>" and adds a learned multiple of the letter's next-token log-probability, so at initialisation the model scores exactly like prompting the base model and training only learns a residual.

Results

TypeSafe public eval (evals.typesafe.ai, rebuilt with the open harness from system-one-open: 20 cases, 372 reference pairs, strict common subset of 343 pairs answered by every published model). None of these four workflows (security incidents, agent-trace observability, invoice processing, customer service) was in the training data.

Training on two unrelated QA datasets moved this benchmark from 63.3 to 76.7. All 372 pairs: 76.6 %. By workflow: invoice 82.6 (184), customer service 78.3 (92), agent trace 70.8 (48), security 56.2 (48). By type: noul 80.9 (236), choice 76.1 (109), score 40.7 (27, a type absent from the training data). With 372 pairs the per-workflow numbers carry roughly ±7–14 points of sampling error.

In-domain (500 held-out items per task, temperature fitted on the same validation set):

Held-out tasks (never trained, temperature not refitted):

Base-model dependence: with the recurrence reduced to 2 steps the prompting baseline drops to ARC 77.5 / BoolQ 77.3, and with 1 step to 57.5 / 37.8. Keep total_ut_steps = 4.

Training

• Data: 2,000 ARC-Easy (choice, 3–5 options) + 2,000 BoolQ (noul) training records, 1 epoch, lr 5e-5, batch 4 × 4 accumulation, gradient checkpointing, ~45 minutes on one RTX 4090.
• Loss: cross-entropy + Brier on the option distribution. Checkpoint selected by calibrated NLL on the validation set.
• Calibration: one global temperature T = 1.3155 fitted on the ARC+BoolQ validation set (ECE 0.027 → 0.013). On out-of-domain data the model is somewhat over-confident (TypeSafe eval ECE 0.078); refit T on a small labelled sample of your own task if you rely on the probabilities.
• Licenses: Ouro-2.6B Apache-2.0; ARC (CC BY-SA 4.0); BoolQ (CC BY-SA 3.0). This adapter: Apache-2.0.

Usage

Requires transformers<4.56 (the Ouro remote code breaks on 5.x), peft>=0.17,<0.18, torch. The base model is downloaded from the Hub with trust_remote_code=True.

pip install "torch" "transformers==4.54.1" "peft==0.17.1" "huggingface_hub"

from inference import DecisionModel

model = DecisionModel.from_pretrained("noscienthoon/ouro-2.6b-decision-lora", device="cuda")   # ~6 GB in bf16
answers = model.decide(
    state={"channel": "email", "subject": "Charged twice", "body": "Two charges for one order. Fix this today."},
    questions={
        "queue":    {"type": "choice", "prompt": "Which support queue should handle this ticket?",
                     "options": {"billing": "Payments, refunds, duplicate charges", "shipping": "Delivery", "technical": "Bugs", "general": "Other"}},
        "priority": {"type": "score", "prompt": "How should this ticket be prioritized?", "levels": ["Low", "Normal", "High", "Critical"]},
        "angry":    {"type": "noul", "prompt": "The customer sounds angry."},
    },
)
print(answers)
# {'queue': {'type': 'choice', 'choice': 'billing', 'probabilities': {...}, 'confidence': ...},
#  'priority': {'type': 'score', 'score': 2.3, 'probabilities': {...}, 'legend': [...], 'confidence': ...},
#  'angry': {'type': 'noul', 'noul': 0.87, 'confidence': ...}}

decide(...) accepts any JSON-serialisable state (a string is used as is, anything else is pretty-printed as JSON), scores each question's options in one forward pass (K sequences for K options), and applies the stored temperature. Pass temperature= to override it. max_query_len (default 4096 here; 512 was used in training, the backbone supports 65k) controls left-side truncation of long states: the end of the state and the question are always kept.

Cost: every option is scored as its own sequence (state + question + options + "Answer: A"), so a question with K options costs K forward passes of the prefix. Ouro's recurrent cache does not support prefix sharing.

Files

Limitations

• Trained on two public QA datasets only; the TypeSafe-eval workflows, score questions and non-English inputs were never seen in training. score is the weakest type (40.7 % on the eval's 27 score pairs).
• Calibration was fitted in-domain and does not transfer perfectly (see above).
• At most 26 options per choice question through the letter readout; larger option sets are scored without the options listed in the prefix.
• The reference answers of the TypeSafe eval are a frontier-model consensus, not human labels.

Related

• jev-ood-calibration: our independent measurement of Jev's calibration in and out of domain (same metrics code as the calibration numbers above).
• system-one-open: the open harness used to rebuild the TypeSafe public eval; its Gemma 4 E2B replica is the 76.7 row in the table.