llama-3.2-3b-sql-lora

Llama-3.2-3B SQL LoRA

LoRA adapter fine-tuning Llama-3.2-3B-Instruct to convert natural language questions + database schema into clean, directly-executable SQL queries.

Model Details

Model Description

This model fine-tunes Llama-3.2-3B-Instruct using LoRA to specialize in text-to-SQL generation. Given a table schema and a natural language question, it outputs the corresponding SQL query with no extra formatting, markdown, or explanation — making it directly usable in automated pipelines.

• Developed by: Bader
• Model type: Causal language model (LoRA adapter)
• Language(s): English (natural language input), SQL (output)
• License: Llama 3.2 Community License
• Finetuned from model: meta-llama/Llama-3.2-3B-Instruct

Model Sources

• Repository: github.com/BY-ALF/n12sql-lora-llama3

Direct Use

Given a schema (as a CREATE TABLE statement) and a natural language question, generates the corresponding SQL query. Intended for prototyping text-to-SQL tools, automating simple database queries from plain English, or as a base for further fine-tuning on domain-specific schemas.

Out-of-Scope Use

Not intended for production use on critical systems without human review of generated queries. Not evaluated on SQL dialects other than standard/SQLite-style syntax. Not evaluated for adversarial/injection-style inputs.

Bias, Risks, and Limitations

• Trained primarily on relatively simple schemas; may struggle with complex, deeply nested, real-world enterprise schemas
• Multi-table joins (3+ tables) are the most common failure mode — occasionally selects incorrect join keys
• No safeguards against SQL injection if used with untrusted input in a live system — always sanitize/validate generated SQL before execution
• Evaluated only via exact-match accuracy, which is a strict metric that doesn't credit semantically equivalent but differently-structured queries

Recommendations

Always review generated SQL before running it against a production database. Consider adding execution-based validation (e.g., running against a sandboxed copy of the schema) before trusting outputs in an automated pipeline.

How to Get Started with the Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "meta-llama/Llama-3.2-3B-Instruct"
adapter_name = "BY-ALF/llama-3.2-3b-sql-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(base_model_name, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base_model, adapter_name)

prompt = """### Context:
CREATE TABLE employees (name VARCHAR, department VARCHAR, salary INTEGER)

### Question:
What is the average salary in the Sales department?

### SQL:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=100, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True).split("### SQL:")[-1].strip())

Training Details

Training Data

b-mc2/sql-create-context — approximately 75,000 examples, each containing a database schema (as a CREATE TABLE statement), a natural language question, and the corresponding correct SQL query.

Training Procedure

Preprocessing

Each example was formatted into a single prompt combining schema, question, and target SQL, separated by section headers (### Context:, ### Question:, ### SQL:).

Training Hyperparameters

• LoRA rank (r): 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Target modules: q_proj, v_proj
• Epochs: 3
• Batch size: 4 (per device), gradient accumulation steps: 2
• Learning rate: 2e-4
• Training regime: bf16 (no 4-bit quantization — trained on AMD ROCm, where bitsandbytes support is limited)

Speeds, Sizes, Times

• Training runtime: ~3 hours 40 minutes
• Steps: 27,993
• Final training loss: 0.80
• Trainable parameters: ~4.6M (0.14% of total 3.2B parameters)

Evaluation

Testing Data

100 held-out examples from the same dataset (b-mc2/sql-create-context), not seen during training.

Metrics

Exact-match accuracy: generated SQL normalized (whitespace, casing) and compared verbatim to ground truth.

Results

Summary

The fine-tuned model substantially outperforms the base model on exact-match accuracy. Manual review showed the base model's underlying SQL logic was frequently correct, but it consistently wrapped output in markdown code fences and added explanatory text, which fails strict exact-match scoring. The fine-tuned model learned to produce clean, directly-executable SQL matching the expected format — a meaningful practical improvement for automated use, though part of the accuracy gap reflects formatting compliance rather than pure reasoning improvement. The most common remaining failure mode is incorrect join conditions in queries involving 3+ tables.

Environmental Impact

• Hardware Type: AMD Radeon RX 7900 XTX (consumer GPU)
• Hours used: ~3.7 hours
• Cloud Provider: None (local training)
• Compute Region: N/A (local)

Technical Specifications

Model Architecture and Objective

Causal language model (Llama 3.2 architecture, 3B parameters) fine-tuned via LoRA for the text-to-SQL generation task, using standard supervised fine-tuning (next-token prediction on schema+question+SQL sequences).

Compute Infrastructure

Hardware

AMD Radeon RX 7900 XTX (24GB VRAM), AMD Ryzen 7 processor

Software

• PyTorch (ROCm 7.1 build)
• transformers, peft, trl, accelerate, datasets
• Ubuntu 26.04 LTS

Model Card Contact

Questions or feedback: open an issue on the linked GitHub repository.