QwerySmith-2.0-dev-seed2

QwerySmith-2.0-dev-seed2

A behaviour-tuned text-to-SQL adapter: it answers questions about a relational database from retrieved evidence, with machine-verifiable citations, and refuses when the evidence does not contain the answer. Facts come from retrieval - this adapter stores no data.

See the results table below.

What it is / is not

• Is: a QLoRA adapter (PEFT) for Qwen/Qwen3-8B teaching the output contract (SQL: / ANSWER: with [table:row_id] citations, or REFUSAL:), schema terminology, and calibrated refusal.
• Is not: a knowledge store. Unmodified Qwen/Qwen3-8B + the same retrieval achieves the fact-bearing part; per the v1.1 finding, fine-tuning taught style, not facts - this release is built on that constraint.

Training recipe (fully pinned)

Training data construction, distractor sampling, and the refusal mix are produced by the harness (triples stage) and are reproducible from the repo.

Dataset

• online_retail_ii - https://archive.ics.uci.edu/dataset/502/online+retail+ii, CC BY 4.0 (UCI Machine Learning Repository)
• 30 questions with gold SQL + expected rows; difficulty-tagged; split mechanically: 10 train_ok / 20 held-out
• Held-out rule: questions whose gold result changes when the last 6 months of data are removed are tagged held-out (cutoff 2011-06-01) and are never used in training. Enforced by a clamped-shadow-database execution check.

Evaluation - measured artifacts only

All numbers below come from captured run artifacts (report.md, failures/), produced by the same harness on identical questions with byte-identical frozen evidence packs for every system.

(Evaluation pending - this card is generated from the training run only. The results table lands here after the eval matrix runs.)

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", device_map="auto")
model = PeftModel.from_pretrained(base, "QwerySmith-2.0-dev-seed2")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

# The model expects the harness prompt contract:
#   SYSTEM contract + SCHEMA (CREATE TABLE text)
#   + QUESTION + RETRIEVED EVIDENCE rows as [table:row_id] lines
# and emits either:
#   SQL: <one SELECT>  /  ANSWER: <text with [table:row_id] citations>
#   or REFUSAL: <what is missing>

Execute generated SQL read-only, with a statement timeout and an AST SELECT-only guard (the harness ships one).

Figures

                    QWERYSMITH 2.0 - ARCHITECTURE & PIPELINE
================================================================================

  RAW CSVs          PREPARE (CPU, local)                TRAIN (T4, Colab)
============      =========================            ==================
 9 Olist tables -> ingest   -> Postgres/SQLite  RAFT   triples jsonl -> QLoRA
                  profile  -> cutoff frozen    TRIPLE       |
                  validate -> leak checks       BUILDER      v
                  retrieve -> evidence packs    ~70/15/15  Qwen3-8B 4-bit
                  triples  -> training data   grounded/   + LoRA r16 a32
                              (train_ok only)  refusal/    lr 1e-4 cosine
                                               schema      3 seeds
================================================================================

  EVAL MATRIX (L4, Colab) - identical questions + identical frozen packs
===============================================================================
   row 1  Qwen3-8B base        + packs --> scorer --> EX / flip / refusal
   row 2  Qwen3-8B + adapter   + packs --> scorer --> mean of 3 seeds
   row 3  Qwen3-30B-A3B AWQ   + packs --> scorer --> on-prem alternative
   row 4  frontier (API)      + packs --> scorer --> reference
                                        |
                     McNemar + GATE (pre-registered) <- report.md + failures/

================================================================================

  INFERENCE CONTRACT (what the fine-tune teaches - behaviour only)
===============================================================================
   prompt = SYSTEM contract + SCHEMA (CREATE TABLE text)
          + QUESTION + RETRIEVED EVIDENCE [table:row_id] rows
                 |
                 v
   SQL:  <one SELECT statement>
   ANSWER: <text with [table:row_id] citations>     or   REFUSAL: <why>
                 |
                 v
   sandboxed execution (read-only role, 30s timeout, sqlglot AST guard)
   facts ALWAYS come from retrieval - the adapter never stores data

Limitations (stated plainly)

• Evaluation is on online_retail_ii only; generalization to other databases is not claimed by this card.
• The adapter emits SQL for the schema it is shown; correctness depends on retrieval quality - the same dependency every system in the matrix has.
• Fine-tuned for the harness output contract; other prompt formats are out of distribution.
• Single-dataset behaviour tune; the 3-seed spread (see sibling repos) indicates run-to-run variance.

Provenance

• Trained by the QwerySmith v3 harness (one-command reproducible): see repository

• Hardware: Tesla T4, 14912 MB VRAM; packages: {"torch": "2.11.0+cu128", "transformers": "5.5.0", "trl": "0.24.0", "peft": "0.20.0", "unsloth": "2026.9.11", "datasets": "4.3.0", "bitsandbytes": "0.50.2"}

• v1.x lineage (1.0/1.1 fine-tuning pipeline and its evaluation): see the repository's legacy/ directory.

License

Apache-2.0 (base model license). Dataset retains CC BY 4.0 (UCI Machine Learning Repository).