Qwen3.5-4B-Hmm

Hmm (Qwen3.5-4B)

Hmm is a small open model that answers questions about your data with probabilities instead of text. You give it a state (any text or JSON) and a few typed questions, it gives back a yes/no probability, a choice or a score.

It is a fine-tuned Qwen3.5-4B. Code, local server and training scripts: github.com/n4ze3m/hmm

Note: This is a fun experiment. Hmm is nowhere near the quality of Jev from TypeSafe and it is not affiliated with TypeSafe in any way. The local server just follows the same request format so it is easy to try.

Files

Usage

Hmm does not generate text, it reads the probability of the option letters on the first token. The easiest way to use it is the small server from the GitHub repo, which does this for you.

1. Start the model

llama-server -hf n4ze3m/Qwen3.5-4B-Hmm:Q4_K_M -np 4 --port 8080

or with Ollama (v0.12.11 or newer)

ollama pull hf.co/n4ze3m/Qwen3.5-4B-Hmm:Q4_K_M

1. Start the server

git clone https://github.com/n4ze3m/hmm.git
cd hmm/server
bun run start

1. Ask something

curl http://localhost:8787/v1/systemone -d '{
  "state": "Help! My payouts have failed for 3 days. I need the money today.",
  "questions": {
    "is_urgent": { "type": "noul", "instructions": "Does this message convey urgency?" },
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Payments, invoicing, refunds",
        "technical": "Bugs, outages, integrations",
        "sales": "Pricing, upgrades, new accounts"
      }
    }
  }
}'

Prompt format

If you want to call the model yourself, this is the exact prompt it was trained on. Take the probabilities of the tokens A, B, C... for the first generated token and normalize them over your options.

<|im_start|>user
State (data to evaluate):
{state}

Question:
{instructions}

Options:
A: {option key} — {option description}
B: {option key} — {option description}
Return only the option letter.<|im_end|>
<|im_start|>assistant
<think>

</think>

For yes/no questions the options are A: false — No and B: true — Yes. For score questions the keys are 0, 1, 2... and the descriptions are your levels, lowest first.

Benchmarks

Note: These numbers are from a few public test sets. Your use case is not one of them, and results can be very different from one use case to another. Hmm does not match Jev on any of them. Please test it on your own data before you trust it with anything.

Everything is zero-shot, none of these test sets were used for training. Majority means always answering the most common label. Qwen3.5-4B is the base model without fine-tuning, same prompt. Jev is only shown where the benchmark's own authors measured it on exactly the same records, I did not run Jev myself.

• On decisions about text you give it (routing, classification, verification, workflows) Hmm is a few points behind Jev. It is not ahead of Jev on anything that counts.
• The fine-tune does not add knowledge, on knowledge benchmarks Hmm is about the same as the base Qwen. On game decisions (maze, doom) Hmm is not useful at all.
• A lot of the score comes from Qwen itself. The fine-tune adds 11 points on the workflow benchmark, about 3 on the Bespoke suite, and it made HellaSwag and phishing slightly worse.
• The Q4_K_M GGUF scores the same as the full model (0.715 vs 0.709 on typed-decisions, inside the noise).

Per-task tables, how the data was rebuilt, the overlap check and the caveats are in BENCHMARKS.md.

Training

• LoRA (r=32) on all attention and MLP projections, 1 epoch, around 30,000 questions.
• The training data is a mix of publicly available datasets and synthetic data. For the synthetic part DeepSeek V4.1 Flash (via Atlas Cloud) wrote around 7,400 cases with 26,000 questions for 100+ workflows, then answered every question again 3 times to get soft labels. Questions where it disagreed with itself were removed. This used about 25M tokens. The synthetic dataset is on Hugging Face: n4ze3m/typed-decisions-synth.
• Training took 2 hours on one rented RTX 3090, which costs about $0.42.
• The loss is plain cross entropy on the option letters, half on the answer and half on the soft labels.

Limitations

• One question is one forward pass, so many questions about a long state are slow.
• The model only knows 26 option letters. Longer lists (up to 255) still work with the server from the GitHub repo, it asks them in chunks of 25, but that is slower and less accurate.
• It was trained on prompts up to 768 tokens and only in English.
• The synthetic training data is written and labelled by an LLM. Nobody checked it.
• It can miss things a careful reader would catch. In my tests it allowed a shell command that hid an rm -rf behind base64. Do not use it as your only safety gate, and do not use it for anything important.

Citation

@misc{nazeem2026hmm,
  author = {Muhammed Nazeem},
  title  = {Hmm: a small open model for typed decisions},
  year   = {2026},
  url    = {https://github.com/n4ze3m/hmm}
}

Support

If you like the project, you can support me on Ko-fi.