text-to-sql-qwen-finetuned

Text-to-SQL Fine-Tuning — Qwen3-1.7B + LoRA

Fine-tunes Qwen3-1.7B with QLoRA to turn a natural-language question + table schema into a single SQL query, and benchmarks it against a GPT-4o few-shot baseline.

Headline result: after ~30 minutes of QLoRA fine-tuning on a single consumer GPU (RTX 3070 Ti, covering only ~40% of one epoch), the 1.7B model outperforms GPT-4o (5-shot prompting) on this task — see Results. Full methodology and caveats are in MODEL_DOCUMENTATION.md.

This repository hosts the LoRA adapter and tokenizer only. The base model, training/eval notebooks, and datasets live in the source project (see Repo layout below, which describes that full project — not everything listed there is hosted here on the Hub).

Model Details

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_path = "Qwen/Qwen3-1.7B"
adapter_path = "prasanthg3/text-to-sql-qwen-finetuned"

tokenizer = AutoTokenizer.from_pretrained(base_model_path)
base_model = AutoModelForCausalLM.from_pretrained(base_model_path, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base_model, adapter_path)
model.eval()

prompt = """<|im_start|>system
You are a Text-to-SQL assistant. Output ONLY a single-line SQL query that answers the question using the given schema.
No explanations, no markdown, no backticks, no preamble.
Rules:
- Use the table name exactly as defined in the schema (often "df").
- Quote identifiers with spaces using double quotes.
- Use single quotes for string literals.
- Refer only to columns present in the schema.
<|im_end|>
<|im_start|>user
Schema:
CREATE TABLE df ("Date" text, "City" text, "Opponent" text, "Results" text, "Type of game" text)

Question:
What type of game was held against France with the results of 3:1?
<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True).split("assistant\n")[-1])

Repo layout

banking77.ipynb      Abandoned/unrelated exploration (GPT-4o intent classification) — not part of this pipeline
fine-tuning.ipynb     QLoRA fine-tuning of Qwen3-1.7B on the Text-to-SQL dataset
eval.ipynb            Baseline (GPT-4o few-shot) + fine-tuned model evaluation, exact-match and LLM-judge
test.py               Downloads the Qwen3-1.7B base model into ./Qwen_model
data/                 train/valid/test parquet splits + saved prediction/judge CSVs
Qwen_model/           Base model weights (downloaded via test.py, not fine-tuned)
trainer_output/       Raw training checkpoints (checkpoint-100 … checkpoint-500)
models/checkpoint-500-best/   Final selected adapter + tokenizer (best eval_loss)

Problem

Generate a correct SQL query from a natural-language question and a CREATE TABLE schema, e.g.:

Question: What type of game was held against France with the results of 3:1?
Schema:   CREATE TABLE df ("Date" text, "City" text, "Opponent" text, "Results¹" text, "Type of game" text)
SQL:      SELECT "Type of game" FROM df WHERE "Results¹" = '3:1' AND "Opponent" = 'france'

Dataset

5,700 natural-language → SQL pairs pooled from ~19 public Text-to-SQL sources (WikiSQL, sql_create_context, Spider, Squall, NVBench, oxenai, sede, criteria2sql, MIMIC-SQL, eICU, ATIS, Advising, Scholar, and others), split into:

Approach

• Base model: Qwen3-1.7B, loaded in 4-bit NF4 (QLoRA) with bfloat16 compute dtype
• Fine-tuning method: LoRA, r=16, alpha=16, dropout=0.05, applied to all linear projections (q/k/v/o_proj, gate/up/down_proj)
• Framework: Hugging Face TRL SFTTrainer + PEFT
• Prompt format: ChatML, ANSI-SQL system instructions + schema + question → single-line SQL
• Training config: 1 epoch target (1,250 steps), effective batch size 4 (per_device_train_batch_size=1 × gradient_accumulation_steps=4), LR 1e-4 constant, gradient checkpointing, load_best_model_at_end on eval_loss
• Hardware: 1× NVIDIA GeForce RTX 3070 Ti (8 GB VRAM)
• Actual run: stopped at step 500/1,250 (~40% of one epoch, ~2,000 training examples seen), ~30 minutes wall clock — the selected checkpoint is the best of a partial run, not a completed one

Results

Evaluated on the 200-example validation set, two ways: strict exact match (normalized SQL string equality) and LLM-judge (GPT-4o rates semantic equivalence — tolerant of aliasing, whitespace, column order, etc.).

The fine-tuned 1.7B model beats the GPT-4o few-shot baseline by both metrics, despite being ~1,000× smaller and trained for well under an hour on a single desktop GPU.

Raw predictions and judge outputs: data/valid_with_gpt4o_predictions.csv, data/valid_with_predictions.csv, data/gpt_judge_results.csv, data/ft_judge_results.csv.

Limitations

• Evaluated on only 200 validation examples — confidence intervals are wide.
• Exact-match is a harsh lower bound (penalizes semantically-identical queries with different formatting); LLM-judge is a closer proxy for real usability but is itself an LLM call and not ground truth.
• The fine-tuning run was manually stopped 40% into one epoch; a completed run may perform differently (better or worse, depending on overfitting).
• The GPT-4o baseline is prompted, not fine-tuned — the comparison is "fine-tune a small model" vs. "prompt a large one," not fine-tuned-vs-fine-tuned.

Reproducing

1. pip install -r requirements.txt
2. Add Azure OpenAI credentials to .env (AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, etc. — see langchain_openai.AzureChatOpenAI usage in the notebooks)
3. python test.py to download the Qwen3-1.7B base model into ./Qwen_model
4. Run fine-tuning.ipynb to train and save the adapter
5. Run eval.ipynb to reproduce the baseline and fine-tuned evaluation numbers

See MODEL_DOCUMENTATION.md for the full write-up.