decider-2b-GGUF

decider-2b: typed decisions with calibrated probabilities in one forward pass

A language model that does not generate text. It reads a state and one or more typed questions, each with an explicit option list, and returns a probability distribution over the options for every question from one forward pass. There is no decoding, no parsing and no output outside the options you defined. It is called from software, not chatted with. It is an open reproduction of the "System One" model class (TypeSafe AI's Jev).

Base model: Qwen/Qwen3.5-2B-Base (1.9B parameters). The supervised stages (v1 to v8) fine-tune it with cross-entropy, a proper scoring rule, on a mixture of about 95 public decision datasets, agent trajectories, web element choice, game states and teacher-written custom questions, in two prompt layouts and with isolated Score levels. This repository holds v10: the v8 weights continued for 384 steps of calibration-aware reinforcement learning whose only rewards are outcomes (live browser task checkers and the exact probability laws of games), with a hard KL limit to the v8 weights on replayed training rows. Code, data registry, training scripts and the recipe are at https://github.com/Mapika/decider; decider/ in this repository is the inference subset of that package. The other sizes and the vision variant are listed under The decider family.

Contents: The decider family · Usage · How it works · Field types · Training · Evaluation · Speed · Limitations · Changelog · Reproduction

The decider family

All six repositories share one interface (decider.infer.Decider, POST /v1/systemone in TypeSafe's format) and one readout: the letter logits at an answer slot, softmaxed over the options. Pick by size and input.

Code, data registry, training scripts, the changelog and the per-version history: https://github.com/Mapika/decider.

Usage

from decider.infer import Decider          # decider/ is included in this repo
d = Decider("Mapika/decider-2b")
d.decide("My card was charged twice for the same purchase.",
         [{"question": "Which department should handle this?", "options": ["billing", "technical support", "sales"]},
          {"question": "Does this need a refund action?", "options": ["no", "yes"]}])
# [{'choice': 'billing', 'confidence': 0.99, 'probs': {...}}, {'choice': 'yes', 'confidence': 0.99, 'probs': {...}}]

decide_batch scores many states, each with many questions, in one call. abstain_below=t returns None for decisions with confidence under t. A question can have 2 to 255 options (more than 10 options use one label token per option, see decider/prompt.py).

The same request shape as TypeSafe's Jev (POST /v1/systemone), in process or over HTTP:

d.system_one({"ticket": {"messages": [{"from": "customer", "text": "I was charged twice for order A-104. Please refund the duplicate."}]},
              "refund_policy": "Duplicate charges are eligible for a refund."},
             {"department": {"type": "choice", "instructions": "Which team should handle this?",
                             "criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
                                          "billing": {"what": "Charges, invoices", "not_for": "delivery"}, "other": None}},
              "refund_requested": {"type": "noul", "instructions": "Does `ticket.messages[0].text` request a refund?"},
              "frustration": {"type": "score", "instructions": "How frustrated is the customer?", "criteria": ["calm", "frustrated", "very frustrated"]}})
# {"model": "decider-v10", "answers": {"department": {"type": "choice", "choice": "billing", "confidence": ..., "certainty": ..., "probabilities": {...}},
#  "refund_requested": {"type": "noul", "noul": ...}, "frustration": {"type": "score", "score": ..., "legend": {...}, ...}}, "usage": {...}}

The state may be a string, object or array (up to 32k tokens with the questions). instructions and every option description may be a string or any JSON value. Question ids are never shown to the model. Each question is scored in its own row, so an answer does not depend on which other questions are asked (independent=False packs them into one row, about half the latency for short states). Each Score level is likewise judged in its own row, without its number or its neighbours, and the per-level fits are normalised ("isolated": false restores listwise scoring). The answer also reports level_fit and their sum fit_mass, which is near 1 when exactly one level fits.

For a fixed set of questions, s = d.schema(questions) computes the question prefix once and s(state) / s.batch(states) then run only the state (1.2 to 2.4x faster per request, up to 19x per batch). It uses a questions-first prompt layout that costs accuracy: about 1.5 points on fixed label sets, 5 on per-example options, more on 50 or more options and on states of several thousand tokens. decider.serve exposes the same thing as POST /v1/systemone; the official typesafe-sdk works against it unchanged with TYPESAFE_BASE_URL pointing at the server.

Requirements: torch, transformers>=5, and flash-linear-attention (Triton kernels for the Qwen3.5 linear-attention layers; the model runs without it but several times slower). Python 3.11 or newer lets those kernels use torch.compile.

Without the helper package, the same computation in plain transformers:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained(REPO); m = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).cuda().eval()
prompt = ("Context:\nMy card was charged twice for the same purchase.\n\n"
          "Question: Which department should handle this?\nOptions:\n(A) billing\n(B) technical support\n(C) sales\nAnswer: (")
ids = tok(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    logits = m(**ids).logits[0, -1]
letters = [tok.encode(L, add_special_tokens=False)[0] for L in "ABC"]
probs = torch.softmax(logits[letters].float() / 1.30, -1)      # -> P(billing), P(technical support), P(sales); 1.30 is the stored temperature

For several questions in one pass, append further Question k: ... Answer k: ( blocks and read the logits at each ( position (see decider/prompt.py).

How it works

The prompt is Context: ... followed by, for each question, the question text, the lettered options (A) ... (B) ... and an answer slot Answer k: (. The hidden state at each slot is projected with the option-letter rows of the LM head and softmaxed over the valid letters, divided by the temperature in decider_config.json. Letters are never generated, so all slots are read from one pass. Large label sets were sub-sampled to at most 10 options per training example (gold always kept, order shuffled), so the model conditions on the supplied candidates rather than on a fixed head.

Field types

• noul: probability of "yes".
• choice with criteria {name: description | JSON | null}: the argmax option, its probability (confidence, the calibrated number), certainty (1 minus the normalised entropy) and the full distribution.
• score with criteria [level descriptions]: the expected level, the probability of the most likely level, the distribution, and the per-level fits.

Training

Supervised stages (v1 to v8). One epoch on a mixture of public decision datasets (intent detection, ticket routing, topic classification, sentiment, emotion, moderation, NLI, paraphrase, fact verification, passage relevance, reading comprehension, multiple-choice QA, ordinal rating scales, pairwise response preference, tool selection), then continuation epochs that added next-action choice from agent trajectories (AgentGym), web element choice (Mind2Web), teacher-written situations and game states, the input shapes of the Jev API (described options, up to 255 options, JSON states with path references, long inputs), teacher- written custom questions with a generic option next to a catch-all, a second cacheable prompt layout, and isolated Score levels. In 10% of questions with three or more options an abstain option is added; in a quarter of those the option list is replaced by labels from an unrelated task so that the abstain option is correct. The full list of components with sizes is in decider/data/mixture.py of the GitHub repository; scripts/train.sh full reproduces the supervised stages in one run.

Reinforcement learning stage (v8 to v10). 384 optimizer steps at a peak learning rate of 1e-6 (cosine, 16 warm-up steps), selected among the checkpoints of a 576-step run. Each of the 48 iterations plays 4 live MiniWoB++ click tasks, 4 minesweeper boards and 4 game boards (a 5x5 grid with a slippery move, draws from bags of known composition), 4 repeats each, through the same one-pass readout that serves requests. Three loss terms use those rollouts: a PPO clipped surrogate (clip 0.2) on the terminal outcome with a leave-one-replicate-out baseline; a proper log score of the model's stated belief about the immediate outcome of its action against the exact law (games, minesweeper) or the realised outcome (browser); and a rendering-consistency term that pulls the model's answer in the other prompt layout and the reversed option order toward its served answer. A fourth term keeps the model where it was: on 8 replayed supervised rows per step, KL(v8 ‖ student) on the served distribution must stay under 0.01 nats on average and 0.05 on any row, otherwise the step drops the reward terms and follows only the KL gradient. Six browser tasks were held out from reward and used for validation only. No gold labels were used. The recipe and every measurement are in docs/RL.md of the GitHub repository.

Evaluation

94 public tasks, original protocol. Large label sets sub-sampled to 10 options; one temperature fitted on in-task data and stored in decider_config.json. "In-task" means the test splits of the training datasets; "held-out" means datasets never seen in training (TREC, BBC news, PAWS, SciQ, Social IQa, StrategyQA, PubMedQA, TruthfulQA, tweet irony, financial sentiment, ADE, MASSIVE scenario, student question categories, Dolly categories, CR reviews, Financial PhraseBank, CommitmentBank, QuALITY, XStoryCloze, RewardBench, Arena preferences, Hermes tool selection, and an abstention probe). ECE is the expected calibration error with 15 bins.

The two "rebuilt set" rows were measured after the data pipeline was rebuilt on another machine: two datasets no longer download (TREC-fine, the game states) and the current mixture adds held-out probes, so that set has 67 in-task and 28 held-out tasks. Its numbers are comparable to each other, not to the rows above. v10 matches v8 on it.

<details> <summary><b>Per-task accuracy / ECE on the 28 held-out datasets, v8 against v10</b></summary>

Per-task accuracy / ECE on the held-out datasets of the rebuilt set, v8 against v10:

</details>

v10 against v8 on the same rows. Every row below is scored by both models on identical inputs and seeds. Intervals are 95% bootstrap or paired intervals.

The browser gain is in the served distribution rather than in the argmax: sampled play improves by ten points, greedy play by under one. Tic-tac-toe and minesweeper play did not change; a 2B model without search loses most of those games either way. The one measured regression is OpenJev, under one point.

Bespoke's public suite (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through system_one as shipped). decider-2b v10 macro 0.704 / micro 0.711; v9 0.701 / 0.711; Nimble-9B 0.748 / 0.759; Jev 1.13.0 0.760 / 0.773 (the last two copied from Bespoke's report). Per-subset numbers, the JevBench public-item comparison (decider-2b v10 is at 1.000 / 0.847 / 0.459 on the easy / standard / hard public items, against Jev 1.13.0 at 1.000 / 0.986 / 0.730) and recordings of both versions on the same browser pages and game boards are in the GitHub README.

Speed

One NVIDIA GH200, bf16, unchanged from v8 (same architecture, readout and temperature). decider.infer.Decider uses shape-bucketed CUDA graphs; the batching server is decider/serve.py. Support-ticket states of about 230 tokens with 3 to 5 typed questions each:

With the schema cache (Decider.schema), 10 described questions on short chat messages run at 11,180 decisions/s in a batch, and one question with 151 options at 19x the full-forward rate. FP8 (e4m3 weights, per-token activation scales) changes accuracy and calibration by less than the evaluation noise.

Limitations

• A 2B model without reasoning. Knowledge-heavy multiple choice (MMLU, MedQA, ARC) improves little over the base model, and a judgment that needs several steps should be split into several questions.
• English only. Calibration is measured on public datasets and teacher-labelled probes, not on your traffic. Check it on your own labels before using confidence for routing.
• v10 continues the v8 weights. The v9 data for terse bucket names (support, help, account next to other) is not in it: on held-out terse-bucket messages v8 chose the generic bucket correctly 59% of the time where v9 reached 86%. Name or describe the generic option as a bucket (general_support, or a description).
• Rules written into the question ("fill if empty, otherwise skip") are not followed at this size. State the decision as a plain question with described options.
• Picking one record out of a long JSON array by position is the least accurate input shape (0.51 with 64 records against 0.70 with one). Address records by key, or let the helper write the index into the array (0.62).
• Full label sets cost accuracy against 10 sampled options: CLINC 151-way 0.88 against 0.98; DBpedia level 2 with 70 labels is the least calibrated case (ECE 0.14).
• Questions packed into one row (independent=False) see the earlier question texts, and reversing their order changes up to 12% of answers. The default path scores each question alone.
• The v10 browser results are on 22 click-only MiniWoB++ tasks: small synthetic pages with the elements listed as text. Typing, scrolling and real websites were not tested. OpenJev accuracy is 0.8 points lower than v8.
• Abstention: a catch-all option ("none of the above", "other", "unsure") is chosen when nothing on offer fits, not when the exact fine-grained label is merely absent. Wordings far from the training data remain the main risk.
• One in-task dataset, tweet_hate (SemEval-2019 HatEval), stays near chance on its test split, whose collection and label definition differ from the training split. The number is reported as measured.

Changelog

The full entries, with the browser and game recordings and the same-rows comparison against v8, are in docs/CHANGELOG.md of the GitHub repository; docs/HISTORY.md has how each stage was trained and measured.

Reproduction

Code, data registry, training and evaluation scripts, the RL recipe and the per-version history: https://github.com/Mapika/decider. Each release is staged with scripts/stage_release.py and uploaded with scripts/upload_hf.py; the previous weights are kept under the tag v8 in this repository.

GGUF

CPU inference is available through dohnuts.cpp, a native C++ port on llama.cpp. It runs the model on CPU, reads the option-letter logits at each Answer: ( slot from the LM head, and serves the same POST /v1/systemone wire format. No GPU or Python runtime is needed.

This conversion is verifiable and structural; it has not been benchmarked against the PyTorch reference. Q8_0 is the conservative default.

git clone https://github.com/DreamBlooms/dohnuts.cpp
cd dohnuts.cpp
git submodule update --init --depth 1
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON
cmake --build build -j --target dohnuts-cli

build/dohnuts-cli --server --port 8080 \
  --model decider-2b-q8_0.gguf --metadata decider.json

Ask one state several questions (same request shape as the PyTorch server):

curl http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' \
  -d '{"state":{"ticket":{"messages":[{"from":"customer","text":"I was charged twice. Please refund the duplicate."}]}},
       "questions":{"department":{"type":"choice","instructions":"Which team should handle this?",
         "criteria":{"billing":null,"technical support":null,"sales":null}},
         "refund_requested":{"type":"noul","instructions":"Is a refund requested?"}}}'

The answer keeps the core fields (type, choice, probabilities, noul, confidence) and adds the model's native confidence and certainty under native, along with legend / level_fit / fit_mass for isolated Score levels.

Rebuild the file from the upstream weights with scripts/build_decider_gguf.sh (convert with --no-mtp; the base config declares an MTP layer but ships no MTP weights). No retraining or merging is involved.