qwen2.5-1.5b-sql-lora-v1

Qwen2.5-1.5B Text-to-SQL (QLoRA fine-tune) — V1

LoRA adapter for Qwen2.5-1.5B-Instruct, fine-tuned on text-to-SQL generation. This is the first version in a series of experiments studying QLoRA fine-tuning on free Colab.

Results

Evaluated on 100 held-out examples from b-mc2/sql-create-context (exact-match after SQL normalization):

What comes next

A follow-up ablation with the code-specialized Qwen2.5-Coder-1.5B-Instruct base — identical LoRA config, identical training data — reached 69.0% exact-match (+8 points on top of V1). See AmbikaSoni/qwen2.5-coder-1.5b-sql-lora.

Training details

• Method: QLoRA (4-bit NF4 quantization + LoRA adapters)
• Base: Qwen/Qwen2.5-1.5B-Instruct
• Data: 2,500 examples from b-mc2/sql-create-context, 1 epoch
• LoRA: r=16, α=32, dropout=0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable params: ~18M (1.18% of total)
• Optimizer: paged_adamw_8bit, LR 2e-4, cosine schedule, 5 warmup steps
• Effective batch size: 16 (4 × 4 grad accumulation)
• max_length: 512
• Hardware: Tesla T4 (Google Colab free tier)
• Training time: ~27 minutes

Loss curve

Training loss dropped rapidly in the first 20 steps (1.44 → 0.86) as the model learned the output format, then plateaued around 0.75 for the remaining 130 steps as it slowly learned SQL patterns from the data.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "AmbikaSoni/qwen2.5-1.5b-sql-lora-v1")
tokenizer = AutoTokenizer.from_pretrained("AmbikaSoni/qwen2.5-1.5b-sql-lora-v1")

prompt = '''### Instruction:
Given the schema, write a SQL query to answer the question.

### Schema:
CREATE TABLE employees (id INT, name VARCHAR, department VARCHAR, salary INT)

### Question:
What is the average salary in the Engineering department?

### SQL:
'''

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=100, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Limitations

• Trained on only 2,500 examples — struggles with complex JOINs, INTERSECT/UNION, subqueries, and multi-table aggregations
• Learned dataset-specific formatting conventions (double quotes for string literals, lowercase values, unicode dashes). May not match your target SQL dialect exactly.
• English only
• Small model (1.5B) — for production use cases, consider larger base models

Author

Fine-tuned by Ambika as a learning project. Part of studying QLoRA and efficient fine-tuning at IIT Bombay.