moodeng-mt-iol-ai-2026

IOL-AI 2026 submission — Variant B (direct-answer lane primary)

Code provenance: moodeng-mt commit 11210fd69ecc48f0a6fe88cdc3bf7632aeb9907e ("Apply direct-answer flip + rows_left-aware program gate"), submission/ tree extracted verbatim via git show, with files relocated so script.py sits at the root of this tree instead of under a submission/ subdirectory — the competition sandbox runs python script.py at the repo root.

This card replaces the Variant A card. Variant A described a pipeline with no direct-answer lane; that description is factually wrong for Variant B. Variant B makes a first-class direct-answer lane the primary solve path.

Relationship to Variant A

Variant A was moodeng-mt commit d519ec0 (arm 2b). Relative to Variant A, Variant B changes only code, and only 4 of the 22 shipped code files:

• moodeng/synthesis/generate.py
• moodeng/synthesis/pipeline.py
• moodeng/synthesis/prompt.py
• moodeng/runner_glue.py

The other 18 code files are byte-identical to Variant A. The weights are unchanged from Variant A — deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, fp16, ~15.2 GB, shipped under model/. Variant B ships no new weights.

What Variant B does differently

Variant B promotes the free-text / direct-answer path from a last-resort degrade into a first-class lane that runs on every puzzle and carries the primary score. Program synthesis becomes opportunistic rather than the default. The behavior is governed by a DirectAnswerConfig dataclass in moodeng/synthesis/generate.py.

Shipped configuration is the dataclass defaults

The competition sandbox passes no environment variables to the scored subprocess, so the DirectAnswerConfig dataclass defaults are the configuration that runs. There is no separate "production config" layered on top. The shipped defaults are the D1-winning A/B arm:

• enabled = True — the direct-answer lane is the default route. enabled=False restores the exact pre-existing (program-lane-first, free-text-as-last-resort) code path byte-for-byte, as an opt-out arm.
• attempt_programs = "off" — program synthesis is not attempted by default. "leftover_budget" and "always" remain available (via MOODENG_ATTEMPT_PROGRAMS) for A/B arms. When a program is attempted and accepted, it still overrides the direct answer for that puzzle — it is strictly better when real — which is why the gated program plumbing is retained even though the shipped default never fires it.
• max_new_tokens = 1024 — used as a ceiling, not an unconditional override. effective_max_new_tokens takes min(max_new_tokens, tier_derived_cap), so the tier ladder can still shrink the budget under time pressure exactly as it does for every other token budget in the codebase. This ceiling fix was required: an earlier unconditional 1024-token override, with no tier-throttling relief, put a 40-row run at 3044.9 s versus the real 1800 s scored cap. The ceiling can only ever lower the effective cap below what the tier alone would allow — it can never make the lane more expensive than the tier ladder intends.
• min_candidates = 2, program_budget_threshold_s = 30.0.

rows_left-aware program gate

When attempt_programs="leftover_budget" is selected, the program lane only fires once budget.remaining() clears program_budget_threshold_s plus a per-remaining-row reserve when rows_left is known (see pipeline._should_attempt_programs). The gate is checked after the direct-answer lane has already spent its real wall-clock, so it naturally reflects true leftover budget with no separate bookkeeping.

Zero-env-var robustness fix

DirectAnswerConfig.from_env uses an _opt_int parser that closes a "zero env var" trap: an absent env var resolves to the dataclass default (1024), rather than silently collapsing the default to None on every no-env-var run (exactly the competition sandbox's condition). A var that is set but empty is an explicit operator opt-out (None, bare tier-derived cap); set but unparseable falls back to the class default.

Runtime contract

• The sandbox invokes python script.py with CWD = this tree's root, offline, on a single T4 16 GB, fp16, under a 30-minute wall-clock cap.
• Reads the hidden test file from /tmp/data/test.csv.
• Writes submission.csv to CWD with columns id, pred (a positional JSON list), and an optional explanation.
• Invariant: every row gets a non-empty pred list of the correct length no matter what fails (graceful degrade to a valid submission).
• Pinned stack: python 3.10, torch 2.4.0, transformers 4.44.1, tokenizers 0.19.1, numpy 2.1.3, accelerate 0.34.2, pandas 2.2.2.
• No jinja2. Prompts are pre-rendered raw strings built by moodeng.synthesis.prompt; tokenizer.apply_chat_template is never called anywhere in this tree.

Weights

model/ at the root of this tree holds the real deepseek-ai/DeepSeek-R1-Distill-Qwen-7B checkpoint (model_type: qwen2, fp16 safetensors shards, tokenizer, configs), unchanged from Variant A (~15.2 GB). The checkpoint is a plain fp16 checkpoint; on the T4 it loads under bitsandbytes NF4 4-bit with fp16 compute (as documented in script.py's sandbox contract and the native-lane run notes), identical to Variant A's load path. runner_glue.resolve_model_dir probes {base_dir}/model/config.json as part of its resolver chain; if no weights resolve, run_model is a no-op and the fallback-only path still produces a valid submission.csv.

Why Variant B — the authorizing measurement

The direct-answer flip was authorized after it won a real-cap Modal T4 A/B.

Measured 12-row A/B on R1-Distill-7B (from the DirectAnswerConfig docstring): the shipped arm D1 (attempt_programs="off" + max_new_tokens=1024) reached chrF-mean 14.29 and a reached-FINAL-ANSWERS rate of 19/55 (34.5%), versus the prior tier-cap-defaults arm D2 at chrF-mean 10.65 / 5/65 (7.7%), versus the pre-direct-answer-lane arm-2b baseline at chrF-mean 4.99.

D1 was then confirmed at 40 rows — roughly 2x the chrF of the previously validated arm-2b configuration, with full per-row coverage (no fallback answers needed). The 40-row confirm is what exposed the unconditional- override wall-clock blocker, which the max_new_tokens-as-ceiling fix above resolves.

Authorizing gate (orchestrator-confirmed): with the ceiling fix in place, the direct-answer flip won a 40-row real-cap Modal T4 A/B at a weighted platform score of 3.344 versus the arm-2b comparator's 2.835 (+18%), finishing in 1516 s wall-clock, under the 1800 s scored cap — the config that authorized shipping Variant B.

Contamination note

This card and the shipped code contain no puzzle text, gold answers, or test rows. All references to "the puzzle task" are generic; no competition content is embedded.