iolai-2026-4b-direct

iolai-2026 4B direct-JSON solver (Qwen3.5-4B + verified-answer SFT)

IOL-AI 2026 submission. Qwen3.5-4B LoRA-tuned (r32, lr1e-4, ep2 by generated-score selection) on 239 expert-verified linguistics-olympiad answer sets to emit ONLY a JSON list of answers. Shipped as Q8_0 GGUF + the v2 solver pipeline (script.py runs offline on one T4 in 30 min; hidden-set score 0.1154 with 3-sample voting).

Prompting contract

• System (exact training string): "You are an expert linguist solving a linguistics olympiad puzzle. Deduce the rules from the data given, then output ONLY a JSON list of the answers to the asked items, in the order asked — no explanation, no markdown fences, just the JSON list of strings."
• User: <problem data>\n\n<asked items> — nothing else. No few-shot (trained without), thinking disabled (enable_thinking=False in the chat template).
• Output: one JSON list of strings, one element per asked item, in order. Parse with a JSON-first chain (last decodable [...] via raw_decode), fall back to line heuristics (see v2/textproc.py).
• Sampling: greedy for n=1; temperature + per-item majority vote for n=3 (N_SAMPLES=3 — worth ~2x exact-match on our runs).

Adding RAG

Structure retrieval as a labeled block INSIDE the user turn, between data and query:

<problem data>

REFERENCE (retrieved; may be irrelevant — ignore if it conflicts with the data):
<retrieved grammar notes / cognate tables / similar solved problem>

<asked items>

Guidelines learned the hard way:

1. Do not touch the system prompt — the model is format-locked to it; contract changes on tuned models collapse delivery.
2. Budget: LLAMA_CTX is 4096 in script.py (raise to 6144-8192; Q8 4B uses ~5 GB, KV is cheap). Keep retrieved text <= ~1500 tokens; prefill costs ~2 ms/tok on a T4 across 90 rows.
3. Retrieval keys that exist in-band: task_type column (translation / match_letters / fill_blanks / num_to_text / text_to_num) and the problem's language name in the header. v2/prompts.py already has an EXEMPLAR_BANK env (JSON {task_type: {user, assistant}}) for typed few-shot retrieval — measured NEUTRAL-to-negative on the tuned model, so prefer inline REFERENCE blocks over exemplar turns.
4. Mark retrieved text as possibly irrelevant (as above): tuned on clean prompts, the model over-trusts unmarked context.
5. Re-validate the 30-min budget after any prompt growth: tokens_per_row x rows / measured tok/s (40.5 on T4 for this GGUF). The failure mode is silent: the pipeline's budget guard truncates generation, killing the JSON tail (EM -> 0).

Env knobs (script.py / v2)

GGUF_PATH, PROMPT_MODE=direct, N_SAMPLES, MAX_NEW=700, LLAMA_CTX, FEWSHOT=0, REFINE_ROUNDS=0 (keep 0: legacy refine prompts fight the JSON contract), TRANSCRIPT.

License: Apache-2.0 (base + adapters). Training data: expert-verified answers from our lab corpus + Neacsu guide (CC BY) + agurung/ioling slices (CC-BY-SA).