iol-ai-2026-solver

IOL-AI 2026 starter submission

This is a minimal starter repo for the IOL-AI 2026 script competition.

Local setup

python -m venv .venv
source .venv/bin/activate  # Windows PowerShell: .venv\Scripts\Activate.ps1
pip install -U pandas torch transformers accelerate huggingface_hub safetensors

Download an offline model into this repo

The challenge eval sandbox cannot download the model at runtime. Download weights once before pushing to your public Hugging Face model repo:

python download_model.py Qwen/Qwen2.5-1.5B-Instruct ./model

Make sure the model license permits redistribution in a public repo.

Smoke test the submission format

python make_mock_data.py
IOL_DUMMY=1 python script.py
python - <<'PY'
import pandas as pd, json
s = pd.read_csv('submission.csv')
print(s)
print(json.loads(s.loc[0, 'pred']))
PY

For a real local model run after downloading weights:

python script.py

Local practice evaluation

Create a practice CSV with id, context, query, task_type, eval_type, and gold. The gold column must be a JSON list string, matching the pred submission format.

Score an existing submission:

python local_eval.py practice.csv --submission submission.csv --details eval_details.csv

Run script.py on the practice CSV first, then score submission.csv:

IOL_DUMMY=1 python local_eval.py practice.csv --run-script

The evaluator reports length match, approximate exact match, and chrF-style character similarity.

Submit

1. Create a public Hugging Face model repo.
2. Upload this folder, including script.py and model/.
3. Enter the repo id in the competition Space.