Slonik-7B-SFT

Slonik-7B-SFT

A PostgreSQL-focused text-to-SQL model, fine-tuned from Qwen2.5-Coder-7B-Instruct via QLoRA on a mix of BIRD, Spider, Gretel, and 2,000 PostgreSQL-specific examples I synthesized for this run.

This is the SFT-only baseline. For better numbers, see:

• Phani-labs/Slonik-7B-GRPO — same SFT base, then 2000 GRPO steps with execution rewards. 38.2% BIRD-PG, 45.2% BIRD-SQLite.
• Phani-labs/Slonik-7B-GRPO-GGUF — GGUF quantizations (Q4_K_M, Q5_K_M, Q8_0) of the GRPO model for Ollama / llama.cpp / LM Studio.

Why I built this

I tested several text-to-SQL models on PostgreSQL-heavy schemas and most handled simple SELECT queries well but struggled once schemas involved JSONB fields, pgvector similarity, full-text search, array operators, or deeper CTE / window-function patterns. This is the SFT pass focused on those cases.

Results

BIRD Mini-Dev PostgreSQL (500-example official benchmark, execution accuracy):

A 7B model approaching GPT-4o performance with pure SFT — and 10+ points above Qwen-32B on the same benchmark.

Difficulty breakdown

• Simple: 48.6%
• Moderate: 29.6%
• Challenging: 19.6%

What it covers

The SFT data emphasizes PostgreSQL patterns that are often underrepresented in generic text-to-SQL datasets:

• pgvector similarity operators such as <->, <=>, and <#>
• JSONB access and containment patterns using ->, ->>, @>, and jsonb_path_query
• full-text search with tsvector, @@, websearch_to_tsquery, and ranking functions
• array operators such as &&, ANY(...), and array_length
• CTEs, lateral joins, CASE expressions, COALESCE, and window functions

Training

Training data (21,847 examples)

• BIRD-SQL train split — 6,601 examples
• Spider train split — 8,034 examples
• Gretel synthetic text-to-SQL PostgreSQL subset — 5,212 examples (filtered)
• PG-Modern custom synthesis — 2,000 examples covering pgvector, JSONB, full-text search, CTEs, window functions, and array operations. Generated via DeepSeek-V4 with sqlglot-based PostgreSQL parseability validation.

Loss curve

Eval loss flattened around 0.29 near the end of training. I did not see obvious overfitting in this run; the train/eval loss gap stayed under 0.10, with final train loss at 0.226.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Phani-labs/Slonik-7B-SFT",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Phani-labs/Slonik-7B-SFT")

schema = """
CREATE TABLE orders (id INT, user_id INT, total NUMERIC, created_at TIMESTAMPTZ);
CREATE TABLE users (id INT, name TEXT, country TEXT);
"""
question = "For each country, total revenue in 2025, ranked. Use a CTE."

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}"}],
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=400, do_sample=False, temperature=0.0)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Example output:

WITH revenue_by_country AS (
    SELECT u.country, SUM(o.total) AS total_revenue
    FROM orders o
    JOIN users u ON o.user_id = u.id
    WHERE EXTRACT(YEAR FROM o.created_at) = 2025
    GROUP BY u.country
)
SELECT country, total_revenue,
       RANK() OVER (ORDER BY total_revenue DESC) AS revenue_rank
FROM revenue_by_country;

Limitations

A few limitations I noticed while manually reviewing harder PG-Modern evaluation examples:

• Drops longer-tail constraints. On questions with four or five requirements, the model often gets the overall structure right but misses one filter (for example, forgets AND p.id != ref_id, or skips a IS NOT NULL check the gold SQL includes).
• Prefers flat SELECTs over deep CTE chains when both are valid. The gold SQL may use three CTEs; Slonik often collapses to one CTE plus subqueries that compute the same thing.
• Inconsistent function variants. May reach for to_tsquery when websearch_to_tsquery would match the intent more closely (or vice versa). Both parse, but semantics differ for multi-word queries.
• Schema-dependent JSON paths. When a JSON field name is not directly given in the schema, the model sometimes guesses (for example, using a 'feature_count' key instead of computing count(*) over jsonb_path_query).

The GRPO variant (Phani-labs/Slonik-7B-GRPO) addresses some of these — particularly the schema-grounding and dialect issues — through execution-based reinforcement learning.

Notes for Blackwell laptops

On my RTX 5080 laptop the vLLM CUDA 13 wheels did not work cleanly with the CUDA 12.x runtime, so I trained through Unsloth's Triton fallback (no flash-attn, no nvcc). AdamW 8-bit produced NaNs within the first 100 steps every time; adamw_torch_fused with LR 1e-5 and grad clipping at 0.5 was what stabilized the run.

Author

Phani

• GitHub: slonik-7b
• GRPO variant (better accuracy): Phani-labs/Slonik-7B-GRPO
• GGUF quantizations: Phani-labs/Slonik-7B-GRPO-GGUF