welsh-text2sql-slms-adapters

Welsh Text-to-SQL with Small Language Models: the adapters

Aled Rowlands, aledcrowlands@gmail.com

These are the trained adapters from my MSc dissertation "Welsh Text-to-SQL with Small Language Models: a Study of Penalty, Recovery and Transfer" (Wrexham University, 2026). Everything else, the Welsh benchmark, the training data, the notebooks, every score and the figures, is on GitHub:

https://github.com/AledCaioRow/welsh-text2sql-slms

An adapter is a small file of extra weights that training adds on top of a downloaded model, leaving the model itself unchanged. Six small models were each trained twice with QLoRA on Spider's training split, once in English and once in Welsh (machine translated), to write SQL for a question about a database. That gives 12 adapters, plus one repeat of one of them with a different random seed. Each is a folder here, named after its run:

The two score columns are execution accuracy on the 2,057 questions of the Spider test split, in Welsh and in English: the adapter's query and the correct query are both run against the database and the answer counts only if the same rows come back. 0.7472 means about 75 questions in a hundred. The Welsh questions are my own translation of Spider, in the GitHub repository under benchmark/welsh. For comparison, the models out of the box scored between 0.3369 and 0.5960 on the Welsh questions and between 0.5187 and 0.7453 on the English ones.

What is in each folder

• adapter/adapter_model.safetensors, the weights (99 to 262 MB)
• adapter/adapter_config.json, the LoRA settings, including the base model the adapter goes on
• adapter/tokenizer.json, tokenizer_config.json, chat_template.jinja, processor_config.json, the tokenizer files saved with the adapter
• adapter/trainer_state.json, trainer_log.jsonl, all_results.json, train_results.json, the loss curve and final numbers
• training_summary.json, everything about the run: model, data file, every setting, library versions, GPU, loss, runtime, when it was saved
• train_log.txt, the raw training log

training_summary.csv at the top is all thirteen runs in one table. Intermediate checkpoints are not uploaded.

How to use one

Download the folder you want, then load it in 4-bit on top of its base model. This is what the evaluation notebook in the GitHub repository does (code/eval and train/LF_eval_colab.ipynb has the full loading function, with a fallback path and the Gemma detail).

from huggingface_hub import snapshot_download

run = "Qwen3.5-9B_welsh_full_lf"
path = snapshot_download("Aled-caio/welsh-text2sql-slms-adapters", allow_patterns=[run + "/*"])
adapter_dir = f"{path}/{run}/adapter"

# with Unsloth (how the adapters were trained and scored)
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(model_name=adapter_dir, max_seq_length=4096, load_in_4bit=True)
FastModel.for_inference(model)

# or with transformers and PEFT
# from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
# from peft import PeftModel
# base = "unsloth/Qwen3.5-9B"   # base_model_name_or_path in adapter/adapter_config.json
# tokenizer = AutoTokenizer.from_pretrained(adapter_dir)
# model = AutoModelForCausalLM.from_pretrained(base, device_map="auto",
#             quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"))
# model = PeftModel.from_pretrained(model, adapter_dir)

The prompt the adapters were trained on never changed. It is one user message with this shape (the schema is the CREATE TABLE statements of the database; the training data on GitHub shows the exact format in its schema_ddl field, and models/prompt.txt there gives it with its hash):

You are a text-to-SQL system. Given a database schema and a question, write one valid SQLite query that answers the question. Output only the SQL query, with no explanation and no markdown.

### Database schema:
CREATE TABLE "clwb" (
  "id_clwb" NUMBER PRIMARY KEY,
  "enw" TEXT,
  "rheolwr" TEXT,
  "capten" TEXT,
  "gwneuthurwr" TEXT,
  "noddwr" TEXT
);

CREATE TABLE "chwaraewr" (
  "id_chwaraewr" NUMBER PRIMARY KEY,
  "enw" TEXT,
  "gwlad" TEXT,
  "enillion" NUMBER,
  "nifer_digwyddiadau" NUMBER,
  "nifer_buddugoliaethau" NUMBER,
  "id_clwb" NUMBER,
  FOREIGN KEY ("id_clwb") REFERENCES "clwb" ("id_clwb")
);

### Question:
Faint o glybiau sydd?

### SQL:

messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Greedy decoding (do_sample=False) and 256 new tokens are what every score above was taken with. The model answers with SQL only; for the example above the right answer is SELECT count(*) FROM clwb.

How they were trained

QLoRA through LLaMA-Factory on Google Colab (one L4, or one A100 for the Gemma runs), 2 to 6 August 2026. One recipe for all thirteen: base model loaded in 4-bit (NF4), LoRA rank 16 and alpha 16, no dropout, learning rate 2e-4, two epochs, effective batch 8, warmup 3 per cent, weight decay 0.01, linear schedule, seed 3407 (3408 for the repeat), 8-bit AdamW, loss on the SQL answer only, sequences cut at 4,096 tokens, training arithmetic in bfloat16, each family's own chat template. Training data: 6,948 rows of Spider's training split, English as released or machine translated into Welsh with NLLB-200. Library versions and everything else are in each training_summary.json.

Licence

CC BY-SA 4.0, the same licence as Spider. The adapters also carry the terms of their base models (Gemma 3, Qwen 3.5, Ministral 3).