sqlpup-394m-sft-grpo

sqlpup-394m-sft-grpo

The SFT model above plus a GRPO stage over an execution reward. 23.51% execution accuracy on BIRD dev.

Pretrained from scratch on 9.67B tokens, supervised fine-tuned, then trained with GRPO against a three-tier execution reward on prompts the supervised stage did not sample. The gain over the supervised checkpoint is significant in one decoding configuration of three, which the project reports rather than reducing to a single number.

Code, configs, and the per-run artifacts behind every number here: github.com/shivenkk/sqlpup.

What it is

A 394M-parameter decoder trained from scratch for text-to-SQL: no pretrained initialisation, no distillation from a larger model. The corpus (9.67B tokens) and the 32,768-entry byte-level BPE tokenizer were built for this task.

Architecture: Llama-style, 1024 wide by 32 layers, grouped-query attention with 16 query heads and 4 key/value heads, RoPE, SwiGLU, RMSNorm, 2048-token context.

Measured accuracy

Execution accuracy on the BIRD development split, all 1534 questions:

The voting rows are means over sampling seeds 0/101/202. Absolute accuracy is far from frontier systems; the point of the project was a controlled measurement of what from-scratch pretraining at this scale buys, not a leaderboard score.

Prompt format

The model expects the schema's CREATE TABLE statements verbatim, then the question, then BIRD's evidence string. It emits a short comment block naming the tables and columns it intends to use, then the SQL. Use the accompanying code so the format and the SQL extraction match what the numbers were measured with.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("shivenkk/sqlpup-394m-sft-grpo")
tok = AutoTokenizer.from_pretrained("shivenkk/sqlpup-394m-sft-grpo")

For evaluation, prefer the project's own harness, which handles the prompt spec, over-context schema compaction, and execution-guided voting:

python -m sqlpup.cli eval predict --model-dir <dir> --examples dev.json \
  --db-root dev_databases --out-dir out --compact-overflow --self-consistency 7

Limitations

• 2048-token context. Large schemas need the compaction pass to fit.
• Trained and measured on BIRD only. No test-split number, no second dataset.
• Roughly 1% of dev questions produce no parseable SQL in the voting configuration, recorded as errors rather than dropped.

Training data

FineWeb-Edu, StarCoder, the Python subset of The Stack, SynSQL-2.5M, and SchemaPile, deduplicated and decontaminated against the evaluation sets.