Downloads · 30 days
10
19% of all-time downloads
richardlian/iolai-2026-4b-direct-vote5
iolai-2026-4b-direct-vote5 is a machine learning model from richardlian. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Hidden-set score: 0.1453 (EM 0.0935, chrF 0.2259, 90/90 rows delivered). That is the best of the ten configurations we submitted, and it beat the 12-July public leaderboard top (0.147 EM 0.2592 / chrF 0.0833) on chrF…
Downloads · 30 days
10
19% of all-time downloads
All-time downloads
54
Public
Repo size
4.9 GB
Likes
0
Public
Click a slice to open those files.
.gguf4.5 GB · 77%
From the Hugging Face model README
Hidden-set score: 0.1453 (EM 0.0935, chrF 0.2259, 90/90 rows delivered). That is the best of the ten configurations we submitted, and it beat the 12-July public leaderboard top (0.147 EM 0.2592 / chrF 0.0833) on chrF by ~2.7× while running a 4B model on a T4.
This repo is a complete, self-contained submission: script.py reads
/tmp/data/test.csv and writes submission.csv, with no network access and no
runtime installs. Everything below is meant to be forked and extended — the
solver is deliberately structured so that adding a step, a prompt, or a retrieval
corpus is a small, local change.
| configuration | score | EM | chrF | what changed |
|---|---|---|---|---|
| 4b-direct-vote5 (this repo) | 0.1453 | 0.0935 | 0.2259 | 5-sample self-consistency vote |
| 4b-direct-vote7 | 0.1402 | 0.0889 | 0.2211 | 7 samples |
| 4b-direct-vote12 | 0.1354 | 0.0843 | 0.2176 | 12 samples, parallel driver |
| 4b-direct-vote3 | 0.1154 | 0.0623 | — | 3 samples |
| 26b-baseline | 0.0883 | 0.0359 | 0.2175 | Gemma-4-26B-A4B, single sample |
| 4b-mix | 0.0866 | — | — | base weights + parser changes |
| 4b-direct (n=1) | 0.0739 | — | — | no voting |
| 4b-grug | 0.0743 | — | — | terse rules-then-answer training |
| 4b-cave | 0.0437 | 0.0139 | 0.1378 | compressed-trace training |
| 26b + parser bundle | 0.0312 | 0.0046 | 0.2098 | see "hard-won lessons" |
The two findings that produced most of the score:
script.py sets a recipe in the environment and calls v2.script.main(). Order of
operations is load-bearing:
submission.csv is written before the model loads — every row filled
with an in-domain token from its own puzzle text. Later stages overwrite it via
write-then-rename. A crash, an OOM, or the kill can never leave a blank row, and
blank rows forfeit the chrF partial credit that is half the metric.Answers are emitted as a bare JSON list (the model is fine-tuned to do this), parsed
by scanning for the last [ that json.JSONDecoder().raw_decode accepts — a regex
cannot do this correctly because ] occurs inside quoted answers. Voting is per
item over a normalization (case, surrounding quotes, trailing period) with the
surface form taken from the first sample that proposed the winner.
vote12 sibling repo, and available here)Voting N times naively costs N prefills. Two facts fix that:
AF_UNIX, socketpair all return EPERM — so llama-server and every HTTP
inference server are unusable. Pipes survive.bin/llama-iolbatch is a small llama.cpp example that speaks JSONL over
stdin/stdout: one request carries n_samples, the driver prefills the prompt
once, clones the KV prefix to sibling sequences with llama_memory_seq_cp, and
continuous-batch decodes them on a unified KV pool. Measured: 160 problems × 12
votes in 464 s in the replica.
The driver can also return the raw-model logprob of every emitted token plus that
token's byte length, which lets the Python side map each answer substring back to the
tokens that produced it and weight votes by per-item confidence. On our local
160-problem fixture this is a real gain over plain majority (paired Δgeo +0.0197,
95% CI [+0.0042, +0.0341]) while plain majority is flat in N. Sibling repos carry
that arm; see v2/confidence.py.
git clone https://huggingface.co/richardlian/iolai-2026-4b-direct-vote5
Layout: script.py (recipe) · v2/ (solver package) · model/ (Q8_0 GGUF) ·
vendor/ (llama-cpp-python + deps, cp310) · bin/ (the batch driver).
Every knob is an environment variable read by v2/config.py:
N_SAMPLES=5 # votes per problem
MAX_NEW=700 # generation cap per sample
VOTE_TEMP=1.0 # vote sampling temperature
VOTE_WEIGHT=none # none|tiebreak|full|bestof (confidence weighting)
VOTE_LP_T=0.35 # weight temperature for the weighted modes
PROMPT_MODE=direct # trained direct-JSON answers
BACKEND=parallel # use bin/llama-iolbatch instead of the bindings
SLOTS=16 # concurrent decode streams
LLAMA_CTX=16384 # must cover SLOTS x (prompt + MAX_NEW)
TIME_LIMIT_S=1800 # the platform's hard kill
MAX_NEW × n_rows ÷ measured_tokens_per_second must fit inside TIME_LIMIT_S.
Measured on a real T4: this 4B at Q8 = 40.5 tok/s single-stream (5.5 s load,
5.0 GB VRAM); Gemma-4-26B-A4B at IQ4_XS = 49.5 tok/s, 14.7 GB. We once shipped a
reasoning-length recipe (~1400 tokens/row) that did not fit, and it scored 0.0 —
every row truncated before its answer. The solver degrades gracefully (it shrinks
MAX_NEW under time pressure rather than abandoning rows), but it cannot rescue a
recipe that was never affordable.
A Pass is anything with a name and run(states, ctx):
class MyPass:
name = "mypass"
def run(self, states, ctx):
for s in states:
if ctx.budget.exhausted(): # always ask the budget first
return
texts = ctx.backend.chat([conv], greedy=False, max_new=256)
s.samples.append({"kind": "solve", "text": texts[0], "meta": None})
Add it to v2/pipeline.py::default_passes() and it inherits transcript logging,
per-pass flushing, crash-safety, and the deadline guard.
There is no network in the sandbox, so retrieval means shipping a corpus in the repo
and querying it locally (a pickled inverted index, a small embedding matrix in
.npy, or plain files). Where the retrieved text goes matters more than how you
fetch it:
LLAMA_CTX must still cover
SLOTS × (prompt + MAX_NEW); the driver is started with --fit off so it fails
loudly rather than silently shrinking the context and starving votes.# v2/prompts.py — extend the user turn, leave SYSTEM alone
def solve_messages(context, query, *, fewshot, reference=""):
user = f"{context}\n\n{query}"
if reference:
user = f"REFERENCE (may help; ignore if irrelevant):\n{reference}\n\n{user}"
return [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]
Good corpora for this task are typological facts, IPA charts, and numeral-system descriptions — knowledge a 4B model lacks and cannot derive from five examples. Since 86% of our residual errors are genuinely wrong answers, buying capability is the right target.
Qwen3.5-4B with
AutoModelForCausalLM gives the vision wrapper (text stack at
model.language_model.*) while adapters trained via unsloth key model.model.*.
PEFT warns about missing adapter keys, merges nothing, and ships the base model.
Three of our submissions were base models before we caught it. Always assert
weights changed (not torch.allclose) and keep a base-model control.explanation column
invites boilerplate; a canned sentence is not an explanation and inflates
"explanation rate" to a meaningless 100%.Qwen3.5-4B (Apache-2.0) fine-tuned with LoRA on 239 IOL-style problems with byte-verbatim answers in a direct-JSON output format, merged and quantized to Q8_0 (≈lossless; the 4B has no memory pressure on a 16 GB T4). Training data was expert-verified reasoning traces from our lab's corpus plus openly-licensed worked solutions; no LINGOLY/LingOly-TOO data was used (their terms forbid training).