ornith-1.5-9b-spreadsheetbench-lora

Ornith 1.5 9B, LoRA fine-tune for SpreadsheetBench agent trajectories

LoRA adapter for ornith-ai/Ornith-1.5-9B, trained by Ambit Labs for the Ylookup x Encode hackathon (SpreadsheetBench). Code, harness, evaluation and the full write-up: https://github.com/Ambit-Labs/hacks-enyl-research (README, "How the Ornith 9B was fine-tuned").

Data

282 multi-turn tool-call trajectories (system prompt, task <id> user turn, assistant tool calls, tool outputs truncated to 6000 characters, ending at the submit call) produced by the same agent harness driven by DeepSeek v4 flash on the SpreadsheetBench verified-400 set, keeping only tasks the evaluator scored as passing. 58 failing tasks and 60 passing tasks were held out. Golden workbooks were never in the training text; they only selected which trajectories to keep.

Recipe

LoRA on all linear projections, rank 32, alpha 64, dropout 0.05; bf16; max sequence length 32768, no packing; 2 epochs; learning rate 1e-4 cosine; batch 1 with 8-step gradient accumulation (72 optimizer steps); gradient checkpointing; loss on assistant tokens only via a custom collator (the chat template has no generation tags). TRL 1.12, PEFT 0.20, transformers 5.16, torch 2.14, one H100, 97 minutes. Train loss 0.58 to 0.55.

Serving

Merge into the base with PEFT merge_and_unload() and serve with vLLM 0.28: --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-auto-tool-choice --max-model-len 32768, thinking off through chat_template_kwargs: {"enable_thinking": false}, temperature 0.

Results

Held out (118 tasks): base 9B 45, this adapter 29, DeepSeek-driven harness 82. Full 400: 0.34 pass rate (the base 9B run of the same 400 at higher concurrency scored 0.19). The adapter makes the model quit the tool loop earlier than the base; the repo README explains the likely causes and the next round.