qwen3-4b-scrabble

Qwen3-4B · Scrabble move picker (QLoRA adapter)

LoRA adapter fine-tuned on top of Qwen/Qwen3-4B to return the highest-scoring legal Scrabble move for a given board + rack.

Task

Given a board (15×15 char grid; . = empty, lowercase = placed blank) and a rack, output one play_move JSON object placing only newly-added tiles:

{"tool":"play_move","arguments":{"placements":[{"row":3,"col":10,"letter":"A"}]}}

Training

• Method: QLoRA (4-bit NF4, double-quant), bf16 compute, paged 8-bit AdamW, gradient checkpointing, completion-only loss (prompt masked to -100).
• LoRA: r=32, α=64, dropout=0.05, targets = q/k/v/o/gate/up/down proj.
• Data: compact re-targeting of Cochon123/scrabble-cot-dataset (rack + best move + score + play_move JSON), ~300 tokens/example. Compact data: Cochon123/scrabble-compact-train.
• Eval prompt format: a 15×15 board grid + rack string (see prompt_format.py). Important: use this exact prompt at inference — the model was not trained on the verbose per-tile JSON prompt.

Evaluation

Scored with the benchmark_scrabble solver (ENABLE lexicon) and the leaderboard metric:

score_pct = 100 * sum(model_move_score) / sum(optimal_score)

2nd place on the public leaderboard (DeepSeek V4 Pro) = 27.8 %.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch, json

BASE = "Qwen/Qwen3-4B"; ADAPTER = "Cochon123/qwen3-4b-scrabble"
tok = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16,
                                             device_map="cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()

board = ["."*15]*15                 # 15 strings of 15 chars
board[7] = ".......CAT......."      # place existing tiles
rack = "AEGHOUV"
msgs = [{"role":"system","content":"Pick the highest-scoring legal Scrabble move..."},
        {"role":"user","content":json.dumps({"rack":rack,"board":board})}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
                               enable_thinking=False)
out = model.generate(**tok(text, return_tensors="pt").to("cuda"),
                     max_new_tokens=120, do_sample=False)
print(tok.decode(out[0][tok(text, return_tensors="pt")["input_ids"].shape[1]:],
                 skip_special_tokens=True))