shisa-de-1

An improved Shisa DE-2 has now been released!

Shisa DE-1

Shisa DE-1 (Decision Engine) is a general-purpose "decision model". It takes one state and a list of named questions and returns one typed answer per question. Instead of an answer text, each answer is read from the next-token distribution at the answer boundary, and restricted to the option letters.

This model was trained from google/gemma-4-26B-A4B-it with softmax cross-entropy over the restricted option-letter distribution at the answer boundary. The loss is the deployment readout: the next-token logits are restricted to the K option letter rows and the cross-entropy targets the correct letter.

The vision encoder is present in the weights and was untouched by training.

Train accuracy rose 0.888 → 0.936 → 0.968 across the three epochs.

Model details

Serving

Python client

The easiest way to use DE-1 is the shisa-de client, which implements the readout contract below and returns typed answers and probabilities instead of generated prose.

pip install -U 'shisa-de>=0.2.0'
export SHISA_API_KEY="your-platform-api-key"  # for the hosted endpoint

You can use this library with our Shisa AI platform or a local server implementing /v1/completions with logprobs and prompt_logprobs, such as vLLM. An arbitrary OpenAI-compatible chat endpoint is not sufficient for this readout.

from shisa_de import Choice, DecisionModel, Noul

with DecisionModel() as de:
    result = de.decide(
        "From: [email protected]\nSubject: Your invoice is overdue\n\n"
        "We could not charge your card. Reply with your full card number, expiry, and CVV "
        "so we can release the pending refund.",
        {
            "asks_for": Choice("What is this message trying to obtain?", {
                "payment_card_details": "payment card details",
                "support_callback": "a support callback",
                "nothing": "nothing; it is a routine notice",
            }),
            "phishing": Noul("Is this message phishing?"),
        },
    )
    print(result["asks_for"])                        # payment_card_details
    print(result["phishing"])                        # ~0.99: probability of yes
    print(result.answers["asks_for"].probabilities)  # calibrated, sums to 1

Choice answers with an option key, Noul with the probability of yes, and Score with the probability-weighted position on an ordered scale. classify is the shorter form for label sets: de.classify(text, {"intent": ["spam", "ham"]}) returns a dict-like result with the chosen label at result["intent"]. Questions in one call run concurrently, and result.usage reports request and token counts. Direct text probabilities are temperature-scaled by default; pass calibrated=False for the raw ones. Overflow scores use the separate, uncalibrated semantics described below.

Point the same client at a local server you started with vllm serve (see Launch): DecisionModel.from_endpoint("http://127.0.0.1:8000/v1", api_key="").

From the shell, shisa-de doctor checks the endpoint, the model id, and the tokenizer's answer boundary, and shisa-de ask --state '<text>' --labels spam,ham makes one decision. Source and full documentation: github.com/shisa-ai/shisa-de.

Choices with more than 26 options

Availability: overflow is included in shisa-de 0.2.0, with readout contract de1-letter-slots-v3. Upgrade with pip install -U 'shisa-de>=0.2.0'. The v0.2.0 source and contract include the tested implementation and replay fixtures. Version 0.1.2 retains the 26-option single-choice limit.

The selected rule splits text choices into balanced consecutive chunks of at most 26 options, keeps each chunk's winner, then asks a fresh choice among those finalists in original order. Ordinary choices with at most 26 options keep their direct readout. The implementation supports 27–676 text options without recursion; live quality testing covered at most 151. Image and ordered-score overflow remain unsupported. overflow="error" retains strict rejection, and max_logprobs=40 can reduce fallback requests on servers that allow it; the SDK default remains 20.

Overflow scores are conditional on surviving the first stage. The full-key map gives eliminated options zero and finalists their final-round softmax. These are not calibrated global probabilities: overflow returns calibrated=False, confidence=None, and strategy/score-semantics metadata. Explicit calibrated=True is rejected for overflow. Option order can change the answer. See the held-out comparison below for measured quality and cost.

On the quality harness's kev-decision-v1 suite (2026-10-02, vLLM 0.30.0 TP1), the rule raises coverage from 504 to 776 of 776 choice questions: this checkpoint scores 0.7423 overall and 0.6507 on the 272 Banking77 questions, against 0.7552 and 0.6618 for the frozen Gemma 4 base. The frozen base leads that suite at this serving shape, while under the 2026-09-23 TP2 stack the two were level (0.7778 each over the 504 answerable questions). Re-running the frozen base under both stacks moves 20 of its 504 narrow questions (4.0%) and this checkpoint 25 (5.0%), so suite deltas of a point or two are inside the serving-shape band.

Readout contract

The direct readout handles up to 26 options in one prompt, with additional requests for missing letter logprobs. Overflow repeats this per-prompt contract across chunks and finalists as described above. Render the state and the question with the option list, then read logprobs at the answer position with max_tokens=1. Restrict the letter logits to the K valid option letters (A upward) and take the argmax for a choice answer or the softmax over those rows for a noul probability. The model emits no answer string, and its unrestricted next-token distribution is not part of the contract.

Example. The initial request for a direct-readout question. Render the prompt with the model's chat template, then read the first generated token's logprobs:

curl http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "shisa-de-1", "prompt": "<rendered prompt>", "max_tokens": 1, "temperature": 0, "logprobs": 20}'

The measured scaffold sends a system instruction and a JSON payload in the user message, then appends the model's generation prompt:

<bos><|turn>system
Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.<turn|>
<|turn>user
{"evidence": "From: [email protected]\nSubject: Your invoice is overdue\n\nWe could not charge your card. Reply with your full card number, expiry, and CVV so we can release the pending refund.", "criterion": "What is this message trying to obtain?", "options": [{"letter": "A", "description": "payment card details"}, {"letter": "B", "description": "a support callback"}, {"letter": "C", "description": "nothing; it is a routine notice"}]}<turn|>
<|turn>model
<|channel>thought
<channel|>

That prompt is 161 tokens. The response (truncated) for its three options is:

{
  "choices": [{"text": "A", "logprobs": {"top_logprobs": [
    {"A": -0.0056, "B": -6.1306, "D": -7.3806, "C": -7.5056, "-": -7.5056, "1": -9.3806}
  ]}}],
  "usage": {"prompt_tokens": 161, "completion_tokens": 1, "total_tokens": 162}
}

Restrict the candidate rows to the option letters and normalize:

The answer is A at 0.9973. Tokens outside the option list are discarded even when they outrank a valid option, as D does here. choices[0].text is the sampled token, shown for debugging only; the contract reads the distribution, not the sample.

A noul question uses the option pair Yes/No, and the normalized probability of the Yes row is the reported noul value. Each option letter must be a single token, and prompt + letter must tokenize as the prompt tokens plus exactly that letter's token. If a letter is missing from top_logprobs, repeat the request with prompt_logprobs: 0 and the letter appended to the prompt, and read the last entry.

Launch

This model can be served via mainline vLLM:

vllm serve shisa-ai/shisa-de-1

On 2 x NVIDIA H20-3e (TP2), fraud-detect suite (59 cases, 295 questions, all splits):

Single-sweep latency at a median 201 input tokens per case:

Training

Data

The base Gemma 4 model is already quite strong, but we trained on 3,632 training examples for specific use cases, including compaction, fraud detection, anti-slop, and safety guardrails.

Training used softmax cross-entropy restricted to the option-letter rows at the answer boundary, so the training objective is the serving readout. A dot-product head over frozen hidden states was also trained and scored lower on the sealed compaction test: 0.837 AUROC at its best epoch against 0.8902 for this model.

Evaluation

The quality harness ran 29 suites against this checkpoint, plus option-order-reversed variants of 13 of them.

While even much smaller classifier models can perform admirably at many decision-making tasks, our testing showed that larger LLMs provide better generalization and can still run within real-time performance windows.

Latencies should be taken as broad guidance. While all models were measured on the same-sized runs, some smaller models were run at TP1, while DeepSeek-V4.1-Flash requires TP4, and Jev is a hosted solution and run remotely and includes network latency to our data center.

More-than-26-option held-out comparison

On 2026-10-02, a paired comparison selected top-one finalist reranking on 100 development questions, froze that rule, and evaluated it on 500 different questions, 100 per benchmark. All measurements used hosted shisa-ai/shisa-de-1, fingerprint vllm-0.26.0-tp2-ae644bcc, with top-k 40. Each prompt retained the de1-letter-slots-v2 readout; the composite protocol is implemented by the SDK's v3 contract.

The capped zero is the refusal policy's outcome. Other entries are measured sample row accuracies, not full-benchmark macro-F1, POP909 song-macro scores, or an overall Decision Index rescore. The measured complete-run Decision Index remains 37.79 under its original local serving and capped readout.

Finalist reranking won on 41 paired rows and lost on 24 versus peak chunks: +3.4 percentage points, with a descriptive 95% bootstrap interval of +0.2 to +6.8 points (2,000 resamples within benchmark, POP909 grouped by song). Two POP909 song IDs appear in both development and held-out samples; rows are disjoint. Costs were 5.028 versus 4.028 HTTP requests per question, including letter fallbacks. These were unbatched HTTP calls, not isolated latency tests.

On 100 further questions, finalist accuracy was 55% in native order and 56% after a fixed permutation, but 26% of chosen answers changed. This rule is not order-invariant, and the experiment does not establish a universally best protocol or calibrated confidence.

Reproducibility: the paired records, development selection, held-out bootstrap analysis, order checks and audit are in the research-jev-universal-classifiers research checkout at commit ef18e24, under evals/results/decision-index-overflow-compare/; the report is evals/reports/DE1-OVERFLOW.md. Those research commits are not yet published; the SDK implementation and five recorded replay cases are public in v0.2.0. A follow-up SDK check replayed five recorded component distributions exactly and matched all five choices on fresh hosted calls; the SDK suite passed 128 offline and 8 opt-in live tests.

Vision suites

The vision tower was left untouched, but on glance's four image-classification suites DE-1 still modestly outperforms the Gemma 4 base (8/12 metrics improved, no regressions). We ran the Qwen3-VL-4B on the matched item set for reference (the vendored Qwen3-VL-4B results were on a different item set, so are not directly comparable).

The letter rows are dashes for Qwen3-VL-4B, where the letter readout errored on every item.

All suites measured on this checkpoint

Metric per suite is the suite's primary metric, and all three models ran the same questions.

* kev-decision-v1 was re-measured on 2026-10-02 with the overflow readout from shisa-de 0.2.0: 272 of its 776 choice questions are Banking77 intent questions with 77 options, which exceed the 26 single-token letter slots this readout restricts to, and they are now answered by chunking plus a finalist round instead of being recorded unsupported. All three models now cover 776/776 questions in this row. Overflow distributions are conditional on finalist selection and are not calibrated; choice accuracy is unaffected.

† kev-transfer-v1 uses the post-fix runs: 0.7823 for this checkpoint, 0.7899 for the frozen base and 0.8103 for hosted Jev, against 0.4914 and 0.5075 before the 2026-09-23 renderer fix that had suppressed option keys carrying no description.

Choice accuracy and option-order sensitivity

Reversal rewrites each choice question's option order and leaves gold keyed by option id, so a position-invariant model scores identically. Deltas are reversed minus original. The original column is also this model's choice accuracy on those suites. These rows are the 2026-10-02 re-measurement of the published checkpoint on vLLM 0.30.0 at TP1 with the overflow readout, so kev-decision-v1 covers all 776 of its choice questions and kev-transfer-v1 carries the post-fix renderer; the previous version of this table came from the 2026-09-23 TP2 run.

Mean absolute shift 0.029. The 11-option counting probe is the worst case, and the chosen-count distribution moves with the presentation: on identical cases and gold, the model picks count 3 on 20 cases ascending and 31 descending. The readout is not scoring the option text alone.

Calibration

A post-hoc temperature is fitted on this checkpoint's dev-split rows by NLL grid search, separately per answer kind because noul and choice calibrate differently: T_noul = 1.69 and T_choice = 1.90 (T_global = 1.84). On the test split (n=7,446), noul ECE moves from 0.054 to 0.042 and choice ECE from 0.138 to 0.046. Accuracy is unchanged, because tempering does not move the argmax. Leave-one-suite-out fits span 1.76 to 1.89, so one temperature transfers across suites. Refit on your own rows before treating the noul probabilities as confidence.

Limitations

1. Held-out decision families remain behind. kev-transfer-v1 scores 0.782 against hosted Jev's 0.810 on the post-fix renderer; the earlier 0.491 figure was the 2026-09-23 rendering defect, which had suppressed option keys carrying no description, not a readout limit. A prompt-scaffold A/B moved nothing.
2. Arithmetic and counting stay in code. jdb-probe-counting 0.640 and jdb-probe-fresh_math 0.617 on these weights, and the failure class appears at every model size tested.
3. Option order changes answers. Reversal costs 11 points on the 11-option counting probe and 5 points on jeff8-ag_news.
4. No generation quality is claimed. Only the restricted-letter readout was trained and measured. Open-ended generation, chat, and tool use are unsupported by the evidence here, and the vision numbers cover only the four glance suites above.