slm-rl-jay-slm-2

jmandava/slm-rl-jay-slm-2

PEFT LoRA adapter that warm-starts Space Invaders play for LiquidAI/LFM2.5-350M in the SLM-RL workshop.

Paste jmandava/slm-rl-jay-slm-2 as the playground adapter URL (and usually the same id as the dataset URL).

Install

pip install "transformers>=4.46" peft accelerate torch

Load with transformers + PEFT

Weights live under the adapter/ subfolder — pass subfolder="adapter".

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "LiquidAI/LFM2.5-350M"
ADAPTER = "jmandava/slm-rl-jay-slm-2"  # this repo

device = (
    "cuda" if torch.cuda.is_available()
    else "mps" if torch.backends.mps.is_available()
    else "cpu"
)
dtype = torch.bfloat16 if device != "cpu" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(BASE)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=dtype)
model = PeftModel.from_pretrained(model, ADAPTER, subfolder="adapter")
model.to(device).eval()

messages = [
    {"role": "system", "content": "You play Space Invaders. Reply with ACTION: <id>."},
    {"role": "user", "content": "Legal actions: 1) NOOP 2) UP\nChoose."},
]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=24, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Download only the adapter files

from huggingface_hub import snapshot_download

path = snapshot_download("jmandava/slm-rl-jay-slm-2", allow_patterns="adapter/*")
# then: PeftModel.from_pretrained(base_model, f"{path}/adapter")

Workshop / SLM-RL CLI

slm-rl evolve --game space-invaders \
  --dataset-url jmandava/slm-rl-jay-slm-2-data \
  --adapter-url jmandava/slm-rl-jay-slm-2 \
  --generations 2

Train metrics (if recorded)

{
  "eval": {
    "episodes": 2,
    "intervention_rate": 0.0,
    "invalid_rate": 0.0,
    "mean_entropy": null,
    "mean_score": 0.5,
    "primary": 0.5,
    "win_rate": 0.0
  },
  "gate": {
    "promoted": true,
    "reason": "primary 0.2500 -> 0.5000, invalid_rate 0.0000, intervention_rate 0.0000"
  },
  "train": {
    "entropy": 0.24262098781764507,
    "frac_reward_zero_std": 0.75,
    "kl": 0.011351149369147606,
    "loss": -0.03604373633861542,
    "num_prompts": 32,
    "reward": 0.6149248331785202
  }
}

Trained with SLM-RL.