llama-3.2-1b-text-to-sql

Llama-3.2-1B Text-to-SQL (QLoRA Fine-tuned)

LoRA adapter fine-tuned via QLoRA + SFT on b-mc2/sql-create-context, converting a table schema + natural language question into a SQL query.

Release date: August 10, 2026

Results (held-out test set, n=5000)

Metric definitions:

• Exact match: generated SQL is identical to ground truth after normalization (lowercase, whitespace, quote style).
• Normalized match: generated SQL has the same SELECT columns, FROM table, and WHERE conditions as ground truth, even if phrased slightly differently (more lenient than exact match).

Usage

This is a LoRA adapter — load it on top of the base model.

from unsloth import FastLanguageModel
from peft import PeftModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-1B-Instruct-bnb-4bit",
    max_seq_length=1024,
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "ssnym/llama-3.2-1b-text-to-sql")
FastLanguageModel.for_inference(model)

SYSTEM_PROMPT = "You are a text-to-SQL assistant. Given a table schema and a question, output only the SQL  Query"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "Context (schema):\nCREATE TABLE head (age INTEGER)\n\nQuestion: How many heads of the departments are older than 56 ?"}
]

inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

outputs = model.generate(inputs, max_new_tokens=128, temperature=0.1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training details

• Base model: Llama-3.2-1B-Instruct, 4-bit NF4 quantization (QLoRA)
• LoRA: r=16, alpha=16, target modules: q/k/v/o/gate/up/down_proj
• Trainable params: 11,272,192 / 1,247,086,592 (0.90%)
• 2 epochs, 73,277 training examples, batch size 32
• Trained on an RTX A5000 (24GB VRAM)

Known limitations

Testing on out-of-distribution examples (custom schemas not from the training dataset) showed strong performance on single-table queries but weaker performance on multi-table JOINs, likely because the training dataset (b-mc2/sql-create-context) is predominantly single-table.

License

Built with Llama.

This adapter is released under the Llama 3.2 Community License, inherited from the base model (Meta Llama 3.2, released Sept 25, 2024). Base model weights via unsloth/Llama-3.2-1B-Instruct-bnb-4bit.

Links

• Notebook: Colab
• Blog post: Fine-Tuning Llama-3.2-1B for Text-to-SQL
• GGUF version: ssnym/llama-3.2-1b-text-to-sql-GGUF