Llama-3.1-8B-text2sql-wikisql

Llama-3.1-8B Text-to-SQL (WikiSQL, QLoRA)

Built with Llama. A QLoRA fine-tune of meta-llama/Llama-3.1-8B for text-to-SQL on WikiSQL.

This LoRA adapter turns the Llama-3.1-8B base model into a text-to-SQL generator. On WikiSQL dev it reaches 0.915 execution accuracy — beating a frontier-cheap API model (GPT-5.4-nano, 0.721) at effectively zero per-query cost (runs locally).

Results — WikiSQL dev (execution accuracy)

Execution accuracy = run the predicted SQL against the real database and compare result sets to the gold query (not string match).

Prompt format

The model was trained on this exact prompt — match it or accuracy drops. The schema is a CREATE TABLE block; generation begins after the final SQL: line.

Construct a SQL statement using the following schema and question:
Schema:
CREATE TABLE players (
  "name" TEXT,
  "school_club_team" TEXT,
  "no" REAL
)

Question: What school did player number 21 play for?
SQL:

Usage

Transformers + PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "meta-llama/Llama-3.1-8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto", load_in_4bit=True)
model = PeftModel.from_pretrained(model, "mohamed-ahmed-58059/Llama-3.1-8B-text2sql-wikisql")

prompt = '''Construct a SQL statement using the following schema and question:
Schema:
CREATE TABLE players (
  "name" TEXT,
  "school_club_team" TEXT,
  "no" REAL
)

Question: What school did player number 21 play for?
SQL:
'''
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

vLLM (serve base + adapter)

vllm serve meta-llama/Llama-3.1-8B --enable-lora \
  --lora-modules text2sql=mohamed-ahmed-58059/Llama-3.1-8B-text2sql-wikisql \
  --max-lora-rank 16
# then request model="text2sql"

Training

• Method: QLoRA (4-bit NF4, bitsandbytes) + LoRA via TRL SFTTrainer, completion-only loss.
• LoRA: r=16, alpha=32, attention + MLP projections.
• Seq length: 256 (examples over this dropped, not truncated).
• Data: WikiSQL, ~54k training examples; 1k validation held out from train.
• Checkpoint: ~step 5800 (lowest validation loss).
• Hardware: single RTX 3080 Ti (12 GB).

Full pipeline (data prep, training, execution-accuracy eval) is open source: github.com/mohamed-ahmed-58059/hf-ml-platform-text2sql.finetune

License

Governed by the Llama 3.1 Community License. Built with Llama.