yes-or-no-1.5b

yes-or-no-1.5b

Answers yes/no questions — and admits when it cannot. It replies with exactly one of nine strings and nothing else: Yes! Yes. Yes? No! No. No? Maybe! Maybe. Maybe? The word is the answer; the punctuation is confidence (! high, . medium, ? low).

Maybe is the point, not a failure mode. An oracle forced to pick a side will tell you that you can give chocolate to your dog. Maybe? means it cannot know; Maybe! means the question is undetermined — a coin toss has no answer, and that is a fact.

It also checks statements as claims, not only questions, refuses non-questions, ignores injected instructions, and obeys an operator policy in the system turn where that policy adds caution — never to overrule a fact. A date given there makes date questions answerable; without one, Maybe?.

Usage

from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
c.chat.completions.create(model="yes-or-no-1.5b", temperature=0,
    messages=[{"role": "user", "content": "Is Berlin in Germany?"}])   # -> Yes!

Only the system turn and the last user message are used; context is 2048 tokens. The chat template ships in the weights and in the GGUF — with llama.cpp pass --jinja.

How good is it, honestly

Held-out, 9,928 rows: answer 89.2%, confidence marker 90.6%, ECE 0.038; in 500 free generations, 0 replies outside the nine strings. Hazardous questions 96.4% — 167 of 167 dangerous ones correct, zero wrong "yes". Speaker-only claims 99.8%, dates 99.2%, world facts 85.0%, contested 70.8%.

Bad at: letters and spelling — does 'banana' have three a is a coin flip (51.7% against 50% for guessing) and confidently wrong, so build nothing on it. Also contested claims, arithmetic past small numbers, anything after 2023, and every language except English.

Trained on 193k purpose-built rows; the dataset is not published. Apache-2.0. Modified weights and chat template; base architecture and tokenizer unchanged — see the model tree for lineage.