Qwen2.5-1.5B-Instruct-text2sql-lora

Qwen2.5-1.5B-Instruct - Text-to-SQL LoRA Adapter

A LoRA adapter fine-tuning Qwen/Qwen2.5-1.5B-Instruct to generate SQL queries from a natural-language question and a database schema.

Results

Evaluated with a custom execution-accuracy metric (does the generated SQL run and return the correct data, checked against an in-memory SQLite database - not a string comparison) over 500 held-out test examples from gretelai/synthetic_text_to_sql:

Breakdown by query complexity - the base model scores 0% on any query needing GROUP BY, JOIN + GROUP BY, or ORDER BY/LIMIT; fine-tuning brings these to 58–76%:

Full methodology, the evaluation code, and training notebook: see the GitHub repository

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model_name = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_repo = "Amr-Kh-2004/Qwen2.5-1.5B-Instruct-text2sql-lora"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(adapter_repo)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto",
    quantization_config=quantization_config,
    torch_dtype=torch.float16,
)
model = PeftModel.from_pretrained(base_model, adapter_repo)
model.eval()

question = "List all employees hired after 2020."
db_schema = "CREATE TABLE employees (id INT, name VARCHAR(50), hire_date DATE);"

chat = [
    {"role": "system", "content": "You are an SQL query generator given a question and a database schema, generate an SQL query."},
    {"role": "user", "content": f"Question: {question}\nDatabase Schema: {db_schema}\nSQL Query:"},
]
prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
input_len = inputs["input_ids"].shape[1]

output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0][input_len:], skip_special_tokens=True))

Training details

• Base model: Qwen2.5-1.5B-Instruct, loaded in 4-bit (NF4)
• Method: LoRA via Unsloth's FastLanguageModel
  • rank 8, alpha 16, dropout 0
  • target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Data: 3,500-example shuffled slice of gretelai/synthetic_text_to_sql's train split
• Schedule: 2 epochs, effective batch size 64 (16 × grad-accum 4), learning rate 2e-4, 8-bit AdamW
• Hardware: free-tier Google Colab (T4 GPU)

Limitations

• Trained on a 3,500-example subset of a 105K+ example dataset - accuracy on JOIN+GROUP BY queries in particular would likely improve with more training data/steps.
• The evaluation metric compares result rows as a multiset (via collections.Counter), so it does not check row order - ORDER BY/LIMIT accuracy may be modestly overstated.
• ~20% of reference queries in the test set fail to execute against the sanitized SQLite schema (Postgres-specific syntax not yet covered by the sanitizer) and are excluded from the accuracy calculation.
• This is a research/learning project, not evaluated for production use, adversarial inputs, or SQL injection safety.