llama-3.2-3b-sql-qlora

llama-3.2-3b-sql-qlora

Fine-tuned meta-llama/Llama-3.2-3B-Instruct on b-mc2/sql-create-context using QLoRA for Text-to-SQL generation.

Results

Evaluated on a 5% held-out split of b-mc2/sql-create-context (seed=42, 3,929 examples).

Training Configuration

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "glen-louis/llama-3.2-3b-sql-qlora")
tokenizer = AutoTokenizer.from_pretrained("glen-louis/llama-3.2-3b-sql-qlora")

messages = [
    {"role": "system", "content": "Given a SQL table schema, write a SQL query that answers the question."},
    {"role": "user", "content": "Table: employees (id INT, name TEXT, department TEXT, salary INT)\nQuestion: What is the average salary by department?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256, do_sample=False)

print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Methodology

What is QLoRA? QLoRA (Quantized Low-Rank Adaptation) combines two techniques: the base model weights are frozen and compressed to 4-bit precision using NF4 quantization (reducing memory from ~6GB to ~2GB for a 3B model), while a small set of trainable LoRA adapter matrices are added in full precision. This means we only update ~20M parameters instead of 3B, making fine-tuning feasible on a single consumer GPU.

Why sql-create-context? The sql-create-context dataset contains ~78,000 natural language question / SQL answer pairs grounded in explicit CREATE TABLE schemas. Each example provides the full table definition, a natural language question, and the correct SQL query — teaching the model to ground its output in real schema structure rather than hallucinate column names or types.

What do the numbers mean? Perplexity measures how surprised the model is by the held-out text under teacher-forcing — a drop from 35.1 to 1.9 means the fine-tuned model assigns substantially higher probability to correct SQL responses. ROUGE-L measures the longest common subsequence overlap between generated and reference SQL on 200 sampled examples — an improvement from 0.909 to 0.986 confirms the model generates more structurally accurate queries, not just lower perplexity.

Limitations

The model is trained for single-table SQL queries matching the style of the sql-create-context dataset. It may struggle with multi-table JOINs, nested subqueries, or dialects (e.g. T-SQL, PL/pgSQL) not represented in the training data. The model has not been RLHF-tuned for safety — use with the same caveats as the base meta-llama/Llama-3.2-3B-Instruct.

Citation

@dataset{sql_create_context,
  author = {b-mc2},
  title = {sql-create-context},
  year = {2023},
  url = {https://huggingface.co/datasets/b-mc2/sql-create-context},
}

Training framework: PyTorch · Transformers · PEFT · TRL · bitsandbytes