QwerySmith-1.1

QwerySmith 1.1 (Qwen3-4B Text-to-SQL)

QwerySmith 1.1 is a specialized, production-ready Text-to-SQL model based on unsloth/Qwen3-4B, fine-tuned using QLoRA via Unsloth.

Benchmark Results

Results

• in_dist: fine-tuned vs 3-shot base, execution-correct only on one side: 13 wins, 0 losses
• gretel_test: fine-tuned vs 3-shot base, execution-correct only on one side: 43 wins, 18 losses
• heldout_sqale: fine-tuned vs 3-shot base, execution-correct only on one side: 4 wins, 2 losses
• heldout_large_schema: fine-tuned vs 3-shot base, execution-correct only on one side: 10 wins, 13 losses

How to read this

• valid SQL: the query runs in SQLite without error.
• exact match: normalized string equality with the gold query. Punishes correct queries written differently.
• execution acc: predicted and gold queries return the same rows. Only items whose gold query returns at least one row are scored (the "scored" column). in_dist has no real data, so tables are filled with random rows seeded from the gold query's literals; that can occasionally make two different queries look equal. gretel_test/heldout_* use each example's own INSERT rows where present (see below).
• If the confidence intervals overlap, do not claim one system beats the other.
• SQLite is not Postgres/MySQL: a few correct queries in other dialects will be marked wrong.
• Base-model output is parsed leniently (code fences and chatter stripped) so it is not punished for formatting.
• v1.1: once a source is part of --mix, its "test" split (e.g. gretel_test) is no longer a generalization check -- it is near-distribution. Only the heldout_<name> sets (sources named in --heldout, which must never also appear in --mix) answer "does this help on SQL styles the model never trained on". Each named source gets its own heldout_<name> set -- compare them individually rather than averaging, since they test different failure modes (SQaLe: real-world schema noise; large_schema: schema-linking pressure, though only its small-schema tail survives the size filter here).
• gretel_test: 79% of items ship with their own INSERT rows; the rest use random filler rows.
• heldout_sqale: 0% of items ship with their own INSERT rows; the rest use random filler rows.
• heldout_large_schema: 0% of items ship with their own INSERT rows; the rest use random filler rows.

How to Use with Unsloth

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained("Cyrax321/QwerySmith-1.1", max_seq_length=2048, load_in_4bit=True)
FastLanguageModel.for_inference(model)

prompt = '''<|im_start|>system
You are a text-to-SQL assistant. Given a database schema and a question, reply with exactly one SQL query and nothing else.<|im_end|>
<|im_start|>user
Schema: CREATE TABLE employees (id INT, name VARCHAR, salary INT, department VARCHAR);
Question: What is the highest salary in the engineering department?<|im_end|>
<|im_start|>assistant
<think>
</think>
'''

inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256, use_cache=True)
print(tokenizer.batch_decode(outputs)[0])