spark-s1-4b-v3

spark-s1-4b-v3

spark-s1 is an open, local "System One" decision model from the Nokast AI community (open-spark-Jev). It answers typed questions about a state (choice, boolean, ordered score, over options defined at request time) with a full probability distribution and a confidence from one forward pass, with no generated text. This is the accuracy pick of the v3 release: Qwen/Qwen3-4B fine-tuned with LoRA r16 (merged into these weights).

It is not Jev and is not affiliated with TypeSafe AI: it is a public backbone fine-tuned with a supervised loss. It follows the same contract (typed questions in; per-option probabilities, confidence out; no decoding).

How it reads an answer

The state is rendered once; each question is appended as a menu (A. ..., B. ..., option definitions in the prompt) followed by the Qwen3 non-thinking assistant header. The logits of the first answer token are restricted to the option-letter tokens (A..Z), divided by a per-type temperature and softmaxed. That distribution is the answer. The weights are an ordinary Qwen3 causal LM; the readout is done by the open-spark-Jev repository (https://github.com/abhishek085/open-spark-jev) code (open_spark_jev.model.MenuScorer, open_spark_jev/serve), which also serves the POST /v1/decide API. calibration.json holds the temperatures: choice 3.166 (fitted on a held-out calibration split); boolean and score are 1.0 (not fitted).

Training

LoRA r16 (merged), 3 epochs, cross-entropy plus a Brier regulariser, on the 967-row os-datagen train split with each choice row also shown in 4 random option orders (4,277 examples). Labels come from code (policy engines, solvers, controlled worlds), not an LLM judge. Data: abhishek085/spark-s1-osdg-v1. No tool-call-risk data was used.

Results (see BENCHMARKS.md B27, B29-B31 in the repo)

The 60-row set has n=60 (one standard error about 5 points) and per-row failures of older models on it were inspected before this ladder, so treat it as a soft final check. Jev's latency is a hosted, network-inclusive recorded number, so speed ratios are not like-for-like.

Limitations

• Trained on under a thousand rows. Weak on agent next-action routing, urgency scoring, retrieval and termination gates, answer sufficiency (2-12 test rows per pack, noisy).
• Vulnerable-code detection is near chance (0.56 on the external suite); prompt-injection accuracy is fine but its probabilities are uncalibrated (binary temperature not fitted).
• Boolean and score probabilities are not calibrated; only the choice head has a fitted temperature. Wrong boolean/score answers can come with confidence near 1.0.
• Answers depend on the state and definitions you give it; it never explains itself. Do not use it as the only gate on high-impact actions. Calibrate on your own labelled outcomes.
• Not RLCD-trained: outcome-based calibration training is planned on a larger dataset.

Roadmap

More data across more task families (including code/data workflows and security), retraining, calibration for every question type, and outcome-based calibration training. Releases are versioned spark-s1-<size>-v<n>.

License

Apache-2.0 (the Qwen/Qwen3-4B backbone is Apache-2.0). The training data was generated and verified with Gemma-4-26B-A4B, Qwen3.6-35B-A3B and Qwen3.6-27B (see the dataset card); check those terms for commercial use.