slmsql-grpo-v2

slmsql-grpo-v2 — GRPO on Text-to-SQL (a documented negative result)

LoRA adapter for Qwen2.5-Coder-1.5B-Instruct, produced by applying GRPO (Group Relative Policy Optimization, outcome-only execution reward) on top of a two-stage SFT Text-to-SQL model.

TL;DR — this adapter does NOT beat its SFT starting point. It is published as an honest, reproducible negative result: outcome-only GRPO gave ~0 Execution-Accuracy gain on BIRD/Spider, even after we fixed the reward- variance problem that is the usual scapegoat. The value here is the finding and the clean telemetry that proves the mechanism worked but the accuracy did not move. For the actual best model, use Anurich/slmsql-qwen2.5-coder-1.5b.

Results

Both deltas are within the ±~4% noise of these slice sizes. Metric = Execution Accuracy (run predicted SQL against the real SQLite DB, compare result rows to gold). Greedy, single inference, no self-consistency/ensembling.

Why publish a null result?

Because it was earned, and it is informative. We ran GRPO twice, under two different conditions, specifically to rule out the obvious confound:

1. Attempt #1 — GRPO on Spider-train. The model was already ~82% on Spider, so most K-sample groups were all-correct → frac_reward_zero_std ≈ 0.7 (70% of groups had zero reward variance → zero advantage → no gradient). Flat EX. This could be dismissed as "you fed it data it had already mastered." GRPO's advantage is (r − mean)/std; an all-same-reward group has std = 0 → no learning signal. This is the textbook GRPO failure mode.

2. Attempt #2 — GRPO on difficulty-filtered data (this run). We fixed the confound with a DAPO-style pass-rate filter: sampled the SFT model K=8× on 1,367 held-out SynSQL questions and kept only the uncertainty band (0.2 < pass_rate < 0.8). Result: 494 band / 502 all-fail / 184 all-pass — a healthy, balanced difficulty split. GRPO on the 494 band then trained with a perfectly healthy mechanism: frac_reward_zero_std fell to 0.05–0.2, reward_std ≈ 0.44, grad_norm ≈ 0.5, gradient flowing every step.

   And the EX still did not move (table above).

Two controlled conditions — flat-reward and healthy-reward — both flat EX. That rules out the reward-variance explanation and points at the method itself: outcome-only GRPO ≈ 0 EX gain on Text-to-SQL. This independently reproduces the Reward-SQL finding (naive outcome-only GRPO 62.6% vs SFT 63.2% — flat-to- negative). The published GRPO gains (~+2.8 EX) require process rewards / a trained PRM, not the outcome-only reward used here.

Training details

• Base policy: two-stage SFT LoRA adapter (SynSQL foundation → BIRD+Spider in-domain grounding), rank 32 / scale 64, all-linear target modules.
• RL data: 494 pass-rate-filtered SynSQL questions (0.2 < p < 0.8), execution-verifiable (ship runnable SQLite DBs).
• Algorithm: TRL GRPOTrainer, 8 generations/prompt, beta=0 (no reference model), lr=1e-6 (cosine decay), temperature=0.8, epsilon=0.2, use_vllm=False. 400 steps, ~2h42m on a single 24–32 GB NVIDIA card.
• Reward (Arctic-Text2SQL-R1 shape, deliberately simple): 1.0 if the SQL executes and the result matches gold; 0.1 if it executes but is wrong; 0.0 otherwise. Plus a tiny 0.2 format reward for a ```sql block. Sandbox: read-only connection + op-budget interrupt for adversarial SQL.

Intended use & limitations

• Use as a research artifact / reference for what outcome-only GRPO does (and does not do) on Text-to-SQL, and as a baseline for stronger RL (process-reward GRPO) or for distillation (the pivot we took after this result).
• Do not expect it to outperform the SFT model. It is functionally equivalent in accuracy and slightly terser in output length (GRPO nudged completions to ~80 tokens).
• Small-slice evaluation (149 BIRD / 200 Spider); differences under ~4% are noise.

Reproduce

# 1. pass-rate filter (build the uncertainty-zone RL set)
python -m slmsql.passrate_filter rs_source.jsonl \
    --adapter adapters/twostage --k 8 --low 0.2 --high 0.8 --name rl_uncertainty
# 2. GRPO on the band
python -m slmsql.train_grpo_cuda rl_uncertainty.jsonl \
    --base-adapter adapters/twostage --name grpo2 --num-generations 8
# 3. eval
python -m slmsql.evaluate bird_dev_150.jsonl --adapter adapters/grpo2 --name grpo2_bird

Code: https://github.com/Anurich/finetuning_qwen