qwen2.5-1.5b-sql-qlora-merged

qwen2.5-1.5b-sql-qlora-merged

Merged fp16 weights of Qwen/Qwen2.5-1.5B-Instruct, QLoRA-fine-tuned for text-to-SQL: given a CREATE TABLE schema and a natural-language question, emit a single SQLite query.

The LoRA adapter alone is at blaze-star/qwen2.5-1.5b-sql-qlora.

Results

Held-out test split, 1,000 examples never seen in training.

Fine-tuning improved exact match by +25.1 points (49.7% -> 74.8%), a +51% relative gain.

Usage

The model expects the chat template with this system prompt — it was trained with it, and accuracy drops without it.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("blaze-star/qwen2.5-1.5b-sql-qlora-merged", dtype=torch.float16, device_map="auto")
tok = AutoTokenizer.from_pretrained("blaze-star/qwen2.5-1.5b-sql-qlora-merged")

SYSTEM = ("You are a text-to-SQL engine. Given a SQLite schema and a question, reply with a "
          "single SQL query that answers the question. Output only the SQL query: no "
          "explanation, no comments, no markdown code fences.")

schema = "CREATE TABLE head (age INTEGER)"
question = "How many heads of the departments are older than 56?"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}\n\nSQL:"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=96, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
# SELECT COUNT(*) FROM head WHERE age > 56

Use greedy decoding (do_sample=False). Sampling hurts exact match on this task.

What the metric actually measures

Primary metric is normalized exact match against the dataset's reference SQL: lowercased, whitespace collapsed, " unified to ', backticks/brackets stripped, spacing normalized around operators and punctuation. It is strict — a semantically equivalent query that differs in alias naming (AS p vs AS T1) or literal quoting (= '15' vs = 15) counts as a miss.

That strictness is the point, but it must be read correctly: the base model already produces largely correct SQL content (token F1 0.925 before any training). Much of the headroom is conformance to this dataset's canonical SQL style, which is exactly what task-specific fine-tuning buys you. To keep that claim honest this project reports three separate baselines rather than one:

The 3-shot baseline is the important control. Three in-context examples raise format compliance to 99.3% — the model stops wrapping output in markdown fences almost entirely — yet exact match moves only to 52.3%. Formatting was therefore not the bottleneck, and gains above that line are genuine SQL-convention learning, not prompt-format cleanup.

Both models receive an identical prompt and identical output post-processing (fence stripping, leading-prose removal, first-statement extraction), so neither is advantaged by the harness. Secondary metrics: order-insensitive token F1 over SQL tokens (partial credit) and format compliance (fraction of raw generations that were already bare SQL).

Data and leakage control

b-mc2/sql-create-context — natural-language question + CREATE TABLE schema -> SQLite query.

The 78,577 raw rows are deduplicated on a SHA-1 of the normalized (question, schema) pair (4 exact duplicates dropped), shuffled with seed 42, and then test is carved off first, before val and train. Splits: 12,000 train / 750 val / 1,000 test.

Split disjointness is asserted at build time and recorded in data/split_report.json:

{"train_test_overlap": 0, "val_test_overlap": 0, "train_val_overlap": 0}

prepare_data.py raises if any of these is non-zero, so a leaking split cannot be trained on. The test split was used only by evaluate.py, never by train.py.

Training

Loss is computed only on the assistant turn, so the model is never rewarded for reproducing the schema or the question.

Quantization: latency, VRAM, and quality

Merged fp16 model re-quantized with bitsandbytes and benchmarked on the same A100. Latency is a single request generating exactly 64 tokens (20 runs after 3 warmups); quality is exact match on the first 300 test examples.

Limitations

• Single-table, synthetic-ish schemas. sql-create-context schemas are small CREATE TABLE statements derived from WikiSQL/Spider. Performance will not transfer directly to large multi-table production warehouses.
• Exact match is style-sensitive. A correct query written in a different but valid style scores zero. Token F1 is reported alongside for this reason.
• No execution-based evaluation. Queries are compared as strings, not run against a database, so semantic equivalence is undercounted.
• 4-bit inference costs accuracy. The base model loses ~8 points of exact match going from fp16 to 4-bit (57.3% -> 49.7%); see the quantization table for the fine-tuned model's own fp16/8-bit/4-bit spread.
• English only, and the model emits SQLite dialect.

Intended use

Converting natural-language questions into SQLite queries over small, explicitly-provided schemas — a component inside a larger system that supplies the schema and validates or sandboxes the generated query. Do not execute generated SQL against a production database without validation; the model can emit syntactically valid queries that are semantically wrong.

Training code

Full, reproducible pipeline: https://github.com/harshb20/qwen2.5-1.5b-sql-qlora

Citation

@misc{qwen25-1.5b-sql-qlora,
  title  = {QLoRA text-to-SQL fine-tune of Qwen2.5-1.5B-Instruct},
  author = {harshb20},
  year   = {2026},
  url    = {https://github.com/harshb20/qwen2.5-1.5b-sql-qlora}
}