connections-rl-sft

connections-rl-sft (1.5B)

LoRA supervised fine-tuning adapter for Qwen2.5-1.5B-Instruct on NYT Connections. Published as the RL warm start and as a documented cautionary result, not as a recommended model. It teaches the answer format without teaching board-grounding, which raises invalid outputs from 32.1% (base) to 74.1%.

Reading the numbers. Groups correct is a mean count on a 0-4 scale: each board has 4 groups, so the SFT value of 0.012 below means 0.012 of 4 groups per board, i.e. 0.3% of groups. The % of groups column is that value divided by 4. Invalid rate is shown as a percentage in the arm-comparison table and as a bare 0-1 fraction in the seed table, matching how each is stored; both rows are labeled with their units. Values quoted from results-analysis/ (pass@k, entropy/KL) are already 0-1 fractions.

Model Details

• Developed by: Jackson Lukas
• Model type: LoRA adapter (rank 16, alpha 32, all-linear) for a decoder-only causal LM
• Base model: Qwen/Qwen2.5-1.5B-Instruct
• Training stage: Supervised fine-tuning
• Language: English
• License: MIT (the base model carries its own Qwen license)

Model Sources

• Repository: https://github.com/jacksonmlukas/connections-rl
• Technical report: report/findings.md
• Full result tables: report/results.md
• Implementation notes: report/implementation_notes.md
• Raw eval artifacts: jacksonlukas/connections-rl-results
• This adapter: jacksonlukas/connections-rl-sft

Part of connections-rl, a two-scale, three-seed study of what verifiable-reward RL actually transfers, trained end to end on free-tier GPUs. This adapter is stage 1 of 2; the GRPO stage that repairs its grounding failure is connections-rl-grpo.

Intended Uses

Direct Use

Primarily useful as the reproducible warm start for the GRPO stage, and as a concrete example of SFT teaching surface form without grounding. It is the shared initialization for all three 1.5B GRPO seed replicates, which is what isolates RL run-to-run variance from SFT variance.

Downstream Use

The adapter and the surrounding harness are intended as a reproducible artifact for research on reward design and reward over-optimization in verifiable-reward RL. The reward, the leakage-aware splits, the evaluation harness with bootstrap CIs and paired tests, and every checkpoint are public so the finding can be re-derived or contradicted.

Out-of-Scope Use

This is a research artifact, not a product. Do not use it to:

• solve live NYT Connections puzzles competitively or to build a puzzle-solving service; held-out solve rate is at or near zero for every arm in this study;
• draw conclusions about the base model's general capability, since these adapters are narrowly specialized and at least one of them measurably degrades the base model's ability on this task;
• perform any high-stakes reasoning task. Nothing here was evaluated for safety, toxicity, factuality or robustness outside the Connections task.

Bias, Risks, and Limitations

This adapter degrades the base model on the metric that matters most here. It more than doubles the invalid-output rate (32.1% to 74.1%) because it learns to emit the answer template while frequently hallucinating words that are not on the board. Mean reward is negative (-0.038) and below the untrained base (0.049). It solves 0 of 162 held-out puzzles.

Study-level limitations that apply to every adapter here:

• One task (NYT Connections) and one reward design. The conclusion is about this reward's structure/semantics decomposition, not about GRPO in general.
• 807 training boards is small for RL. Memorization is a plausible consequence of data scale as much as of the algorithm.
• Checkpoint-level analyses (entropy, KL, phase transition) come from seed 0 at 7B only, because only that run was Hub-synced during training. Endpoint claims carry n=3 seeds per scale; the timing of the collapse carries n=1.
• NYT Connections boards encode US-centric cultural and idiomatic knowledge, so performance is not representative of word-association ability in general.

Recommendations

Use the GRPO adapter instead if you want the format discipline without the hallucination, or use the base model if you want grouping ability. This adapter is published for reproducibility of the two-stage pipeline.

How to Get Started

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct", dtype=torch.float16, device_map="auto"
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = PeftModel.from_pretrained(base, "jacksonlukas/connections-rl-sft")
model.eval()

words = ["HAIL", "RAIN", "SLEET", "SNOW", "BUCKS", "HEAT", "JAZZ", "NETS",
         "OPTION", "RETURN", "SHIFT", "TAB", "KAYAK", "LEVEL", "MOM", "RACECAR"]
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Group these 16 words into 4 groups of 4:\n"
       + ", ".join(words)}],
    tokenize=False, add_generation_prompt=True,
)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=256, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

The exact prompt template, parser and reward used in the paper are in src/connections_rl. Serving the adapter behind vLLM (--enable-lora) is supported via docker compose up in the repo.

Training Details

Training Data

1,078 NYT Connections puzzles from the gvc-local tagged puzzle database, split strictly chronologically so that every evaluation puzzle postdates every training puzzle:

The chronological split is the leakage control: NYT Connections boards are published daily and widely discussed online, so a random split would let a model benefit from puzzles whose answers circulated before its pre-training cutoff.

Reward Function

Deterministic and unit-tested, with no learned reward model:

Only the structural component is cheaply verifiable per sample. The study's central finding is that GRPO optimizes that component at the expense of the semantic component, which is the part the reward cannot check directly.

Training Procedure

Rank-16 LoRA (alpha 32, all-linear target modules) over the instruct base, 3 epochs on the 807-puzzle train split, completion-only loss on the answer block. Single free-tier T4.

Evaluation

Testing Data, Factors and Metrics

Held-out test split of 162 puzzles (2025-12-15 to 2026-05-29), strictly after every training date. Greedy decoding via vLLM. Bracketed values are percentile bootstrap 95% CIs over 1,000 resamples. Comparisons between arms use exact McNemar tests on solve rate and paired bootstrap on per-puzzle reward. Results are stratified by puzzle category (wordplay, cultural, category, tag-fillin, silent-letter) in the underlying JSON.

Numbers below come from session A (main run). See Reproducibility.

Results

The SFT baseline above is the session A measurement (groups correct 0.012). The seed-replicate cards report 0.019 for the same adapter, measured in session B. Both are correct: greedy decoding is not bitwise deterministic across vLLM layouts. See Reproducibility below.

SFT alone is harmful at this scale. It teaches the output format but not board-grounding, so invalid outputs rise sharply and mean reward drops below the untrained base. The GRPO stage that follows repairs exactly this failure.

Seed replication (1.5B, 3 GRPO seeds). Measured in session B:

Under pass@16 sampling (temperature 0.9, best-of-k scoring): base 0.000 solve / 0.011 groups (fraction), SFT 0.000 / 0.014, GRPO 0.000 / 0.002.

Reproducibility and Measurement Noise

Some arms were measured in two vLLM serving configurations, referred to in the repository as session A (results/, results-7b/) and session B (results-seeds-*/, aggregated in results-seeds/seed_summary.json).

Greedy decoding is not bitwise deterministic across vLLM batching and parallelism layouts, so a small number of borderline tokens flip. The 7B SFT arm differs on 2 of 162 puzzles for grouping and 3 of 162 for format validity between sessions; every GRPO arm reproduces exactly (0 of 162 on all metrics).

That asymmetry is itself evidence for the entropy-collapse account: the final 7B GRPO policy has a measured entropy of 0.0099 nats/token and therefore has no borderline decisions left to flip, whereas the SFT policy has the highest entropy of any arm (0.303 nats/token). Statistics are never mixed across sessions within a single comparison.

Environmental Impact

Trained on free-tier NVIDIA T4 GPUs (16 GB, Turing, pre-Ampere) via Kaggle and Google Colab. Total training compute for this adapter was approximately about 1 hour on a single T4. The entire 1.5B study, including all seed replicates and evaluation, was run at $0 marginal cost on free-tier hardware. Carbon emissions were not directly measured; the T4 has a 70 W TDP, which bounds the energy use of a single run well below 1 kWh.

Citation

If you use this adapter or the accompanying analysis, please cite the repository:

@software{lukas_connections_rl_2026,
  author = {Lukas, Jackson},
  title  = {connections-rl: What Verifiable-Reward RL Actually Transfers},
  year   = {2026},
  url    = {https://github.com/jacksonmlukas/connections-rl},
  note   = {Two-scale, three-seed GRPO study on NYT Connections}
}

The predecessor multi-agent work is published as Snap Out of It (ACL 2025, REALM Workshop).

Model Card Contact

Open an issue at https://github.com/jacksonmlukas/connections-rl/issues.