qwen2.5-3b-text2sql

Qwen2.5-3B Text-to-SQL

A fine-tuned version of Qwen2.5-3B-Instruct that converts natural-language questions into SQL queries, given a database schema. Trained with LoRA/QLoRA on a single free GPU.

• Developed by: ashishsahu2008
• Base model: unsloth/Qwen2.5-3B-Instruct
• Fine-tuned on: b-mc2/sql-create-context
• License: apache-2.0
• Live demo: https://huggingface.co/spaces/ashishsahu2008/text2sql-demo
• Code / write-up: https://github.com/ashishsahu2008/fine-tuning

What it does

Give it a CREATE TABLE schema and a plain-English question, and it returns only the SQL query that answers the question — no explanations or markdown, just runnable SQL.

Results

Evaluated on a held-out test split (n=200) the model never saw during training. Both the base and fine-tuned models were scored with identical string normalization (lowercase, strip ;, canonicalize quotes), so the comparison is apples-to-apples.

Only 0.96% of parameters (~30M of 3.1B) were trained via LoRA. Much of the gain comes from the base model learning to emit clean, executable SQL instead of wrapping answers in prose or markdown fences.

Note on the metric: exact-match undercounts semantically-correct queries that are written differently (column order, aliases, whitespace). A stronger evaluation would use AST comparison (sqlglot) or execution accuracy on a benchmark with populated databases (Spider / BIRD); sql-create-context ships schemas only.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ashishsahu2008/qwen2.5-3b-text2sql"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)

schema = "CREATE TABLE employees (name VARCHAR, salary INTEGER, department VARCHAR)"
question = "List the names of employees in Sales earning over 50000."

messages = [
    {"role": "system", "content": "You are a SQL expert. Given a database schema "
                                   "and a question, output only the SQL query that answers it."},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Prompt format

The model expects the schema and question in a single user turn, with this system prompt:

System:  You are a SQL expert. Given a database schema and a question, output only the SQL query that answers it.
User:    Schema:
         <CREATE TABLE ...>

         Question: <your question>

Training details

• Method: supervised fine-tuning (SFT) with LoRA on a 4-bit quantized base (QLoRA)
• LoRA config: rank 16, alpha 16, applied to attention + MLP projections
• Data: 3,000-row subset of b-mc2/sql-create-context (2,700 train / 300 test)
• Hyperparameters: 2 epochs, learning rate 2e-4, effective batch size 8
• Hardware: single Colab T4, ~25 minutes
• Frameworks: Unsloth + TRL SFTTrainer

Limitations

• Trained on single-table CREATE TABLE schemas; complex multi-join databases are out of distribution.
• Assumes the schema is provided in the prompt — it does not know any tables you don't give it.
• SQL dialect follows the training data (broadly SQLite-compatible); it does not target a specific engine's extensions.

Fine-tuned with Unsloth and Hugging Face's TRL library.