Qwen3.5-9B-Agentic-GRPO

Qwen3.5-9B TextSQL — Agentic (CoT SFT → GRPO)

A 9B text-to-SQL model trained to answer by querying the database, not by translating a question into SQL in one shot. It proposes a query, runs it, reads the rows that come back, and revises — averaging 3.31 tool calls per question on Spider and 4.34 on BIRD, with 100% of questions using at least one call.

Training was chain-of-thought SFT on GPT-distilled reasoning traces, followed by GRPO with GPT-4.1 as the reward model. See TRAINING.md.

Read § Important caveat before using the merged weights. They are the published adapter applied to stock Qwen3.5-9B, which omits two earlier training stages. The benchmark numbers below were measured on exactly these weights, so they describe what you can download — but they are a floor, not the pipeline's ceiling.

---

Results

Evaluated with the agentic loop in eval/agentic_sql.py, --max-turns 8, temperature 0.0, bf16, on a single A100 80GB.

Both runs produced a parseable final query for every single question (1034/1034 and 1534/1534) and used at least one tool call on every question.

BIRD by difficulty

Accuracy vs. number of tool calls (BIRD)

This is selection, not causation — the model keeps probing when a question is hard, so call count is a difficulty proxy. But the 8-call bucket (210 questions at 25.2%) is different in kind: those questions ran out of turns. Combined with forced_final scoring 28.0% against final_sql's 65.7%, BIRD's 8-turn budget is binding and raising it should help. On Spider the ceiling barely binds (57 forced finals), so the two datasets want different budgets.

Spider 2.0-lite (local subset)

Scored with the upstream evaluation_suite/evaluate.py against Spider 2's gold result CSVs.

This is the 135-instance local* subset, not all 547. The other 412 instances (bq, ga, sf) need a paid Google Cloud billing account and a Snowflake access request, and were not run. These numbers are not comparable to a full-set leaderboard entry.

Both Spider 2 runs also use raised capacity limits relative to the Spider 1 / BIRD runs above - --max-new-tokens 2048 (from 1024) and --max-input-tokens 30000 (from 20000) - because 7.5% of generations hit the 1024-token cap mid-generation on Spider 2, where reasoning is much longer. Those are harness limits, not prompt or model changes.

The column metric is inverted

Spider 2.0 scores column selection in the opposite direction to Spider 1: compare_pandas_table walks the gold columns and requires each to be matched by some predicted column. An extra predicted column is free; a missing one is fatal. The prompt used for the Spider 1 / BIRD numbers above encodes the Spider 1 convention ("return ONLY the columns the question asks for"), which can only lose points here. --col-policy spider2 inverts that one block. --col-policy v3 is byte-identical to the prompt used for every other number on this card.

Failure decomposition

The two changes were made in a single run, so +9.6 points cannot be cleanly attributed between them. The deltas suggest roughly one third prompt (too_few_columns halved) and two thirds turn budget (row_count_mismatch and sql_error both fell; max_turns fell 9 to 2 and final_sql rose 52 to 87).

value_mismatch rose, which is the expected shape of progress: instances that previously failed structurally now run and return the right shape with wrong numbers. The residual moves from configuration toward reasoning.

As on BIRD, the stop reason predicts accuracy - final_sql 43.7% against forced_final 19.6%. But raising the cap did not rescue the grinders: the 44 instances that consumed all 20 calls scored 18.2%, essentially matching the 16.9% seen at the 12-call cap. Short runs remain the accurate ones (5-9 calls: 50-86%). Past 20 turns looks like diminishing returns.

Note for anyone reproducing this

evaluation_suite/evaluate.py's get_sqlite_result has no per-query timeout, and its timeout argument only bounds the thread future - which cannot cancel a query already running inside SQLite's C loop. One pathological prediction wedged the evaluation at 134/135 for 9.5 hours and wrote no results. A set_progress_handler watchdog fixes it, since that callback is invoked from inside the loop; see eval/spider2_sqlite_timeout.py.

---

---

Important caveat: the base model

adapter_config.json records the adapter's base as /workspace/models/grpo_qlora_hard_merged — a private intermediate artifact, not stock Qwen3.5-9B. That model was unavailable when these weights were built, so:

published  =  Qwen/Qwen3.5-9B         + Δ_stage3
intended   =  grpo_qlora_hard_merged  + Δ_stage3
missing    =  the CoT-SFT and first-GRPO stages

A LoRA is a delta optimized against particular base weights; applied to a different base it yields a working model, but not the intended one. The adapter itself is unmodified and correct — it ships as adapter/checkpoint-1150-adapter.zip precisely so the intended model can be reconstructed by merging onto the correct base. Details in TRAINING.md § 5.

---

Usage

The model is trained for a multi-turn loop with a SQL execution tool. Used as a one-shot translator it will underperform these numbers — it expects to be able to look things up.

Tool calls are XML, defined by the bundled chat_template.jinja. JSON-style tool calling will not parse:

<tool_call>
<function=execute_sql>
<parameter=query>
SELECT DISTINCT country FROM singer;
</parameter>
</function>
</tool_call>

Run the reference harness:

python eval/agentic_sql.py \
  --model VikramPal/Qwen3.5-9B-Agentic-GRPO \
  --dataset spider --n 1034 \
  --batch-size 16 --max-turns 8 \
  --out spider.json --ckpt spider.partial.json

The full system prompt and few-shot examples are in prompts/prompt_v3.py; the loop design, tool-call parsing, and the safety rules for executing model-written SQL are documented in AGENTIC_FLOW.md.

---

Repository contents

eval/results/ is included so every claim here can be recomputed rather than taken on trust.

---

Limitations and honest reporting

The Spider number is prompt-contaminated; the BIRD number is not. The prompt was refined by analysing failures on Spider dev and then scored on Spider dev. BIRD was never used for prompt development, so 59.9% is the cleaner read on generalization. Spider test.json was left untouched and would give an uncontaminated Spider figure.

Prompt gains were small and not statistically significant. Replacing the previous prompt moved Spider 79.8% → 81.3% under an identical scorer, but that net +16 came from 50 questions fixed and 34 broken — a sign test gives p = 0.10. The run also raised the turn budget from 5 to 8 at the same time; splitting the flips attributes roughly +9 to the extra turns and +7 to the prompt. Neither component is significant alone. Prompt edits on an agentic loop move many answers in both directions, so report fixed/broken/net, not just the delta.

Execution accuracy understates and overstates in different places. On an equivalence analysis of Spider dev:

• Only 18.6% of correct predictions matched the gold SQL as text. Roughly 77% of correct answers are written differently from the reference — exact text match would understate accuracy by about 4×.
• Conversely, ~42% of correct verdicts rest on a single scalar value and ~3% on both queries returning empty, where matching is weak evidence of equivalence.

Some Spider golds are wrong, and the model is penalized for being right. In flight_2, values are stored with whitespace padding (' AHD'). The gold query filters on the unpadded string and returns zero rows; the model probes, matches what is actually stored, returns the correct rows, and is scored wrong. 23 questions (11% of Spider failures) are this pattern — crediting them puts Spider near 82–83%.

Other limitations. SQLite dialect only. Trained on Spider-1-style schemas; very wide or deeply denormalized schemas are out of distribution. Executes model-written SQL, so it needs a read-only connection and a query timeout (see AGENTIC_FLOW.md § 4) — a cross join on a large table will otherwise hang indefinitely.

---

Citation

@misc{qwen35-9b-textsql-agentic-grpo,
  title  = {Qwen3.5-9B TextSQL Agentic: CoT SFT and GRPO for multi-turn text-to-SQL},
  author = {Pal, Vikram},
  year   = {2026},
  url    = {https://huggingface.co/VikramPal/Qwen3.5-9B-Agentic-GRPO}
}

Built on Qwen/Qwen3.5-9B (Apache-2.0). Evaluated on Spider and BIRD.