RaifuWars-RL-ActionScorer

Raifu Wars — RL Action Scorer

A 57,730-parameter policy that plays a seat in Raifu Wars, a turn-based strategy game, through the Warrior protocol.

It wins 55.1% of its matches against the game's own built-in AI, which wins ~28% of its own seats in the same matches. For scale, four equal players in a free-for-all each win 25%.

It also accepts 100% of its tier-ups (309/309) — identical to the built-in AI — where the 4B fine-tune declined 61% of them. Reaching tier 4 is the only win condition in this game, so that one number explains most of the table.

0 illegal actions across 9,540 decisions. Not a training result: the policy scores the legal actions it is offered and takes an index into that list, so an illegal action is unrepresentable rather than merely unlikely.

The interesting part is not the win rate

It was trained almost entirely in a reimplementation of the game (raifusim, in Hemlock) that runs ~1,100× faster than the real thing — 1,547 agent decisions/sec against ~1.4. A conformance harness replays 24,200 real recorded decisions through the sim and reports where the two disagree. Per-map win rate in the real game tracks that conformance:

Four points, monotonic, with a known mechanism for the outlier: Arboretum is the only board whose vegetation is scattered procedurally at match start, so it is match state the map file cannot encode and the sim generates its own. Fidelity predicts transfer, and the one place the simulator is wrong is the one place the policy fails. Excluding Arboretum: 82/119 = 69%.

Architecture

Two towers and an interaction term. The state is embedded once, each offered action is embedded, and the score is head([s, a, s*a]), softmaxed over exactly the actions offered.

state  (33 features) -> 128 -> 64 ┐
                                  ├-> [s, a, s*a] -> 128 -> 1  -> softmax over N offered
action (26 features) -> 128 -> 64 ┘
value head: 64 -> 128 -> 1        (PPO critic; also useful as "how am I doing")

Why scoring rather than classifying. The legal set runs from 2 to ~670 actions and changes every decision with board size, dice roll and hand. A fixed output head would need an entry for every tile on every map, mask nearly all of them every step, and learn nothing transferable between a 17×21 board and a 27×27 one. Scoring makes the same weights work on any board and makes masking free.

Hand-crafted features, deliberately. 33 state + 26 action, all computed from structured fields and never from the English note text the game attaches to each action. Distances are divided by board span so boards of different sizes produce comparable numbers.

Usage

The checkpoint alone is not enough — it needs the feature encoders, which are included here.

import torch
from raifuwars_rl.features import encode_state, encode_actions
from raifuwars_rl.policy import ActionScorer

net = ActionScorer()
net.load_state_dict(torch.load("raifuwars-actionscorer.pt", map_location="cpu")["model"])
net.eval()

# `state` and `actions` are the /v1/act request body of the Warrior protocol
with torch.no_grad():
    s = torch.tensor(encode_state(state))
    a = torch.tensor(encode_actions(state, actions))
    probs = torch.softmax(net(s, a), dim=0)
chosen = actions[int(probs.argmax())]["action_id"]

serve.py exposes it as a Warrior sidecar, which is how the numbers above were measured:

python serve.py raifuwars-actionscorer.pt --port 8899

How it was trained

1. Behaviour cloning on 77,915 recorded decisions from the built-in AI (dataset) — 68.7% held-out agreement, 19.4% win rate.
2. PPO in the sim, warm-started from that. 1.65M decisions in ~36 minutes.

The warm start is not optional. At ~1.4 decisions/sec in the real game, discovering by chance that the way to win is to walk to a specific square and press one particular button is not something a night of exploration finds.

Reward: the tier ladder pays 1/2/3/5 for tiers 1–4 and winning pays 10, with a small shaping term for progress toward the next threshold that saturates — stars you already have enough of are worth exactly zero. That saturation matters: an LLM fine-tune on this game accumulated 838 stars a match, reached tier 2.05, and won twice in forty. It had found the means and never spent it. Verified before training by replaying 360 recorded matches through the reward: the winner out-earned the losers in 360/360, Spearman 0.851 against final tier.

Limitations

• Arboretum, 13%. See above. Known cause, not yet fixed.
• No cards. The sim implements the core loop and not the 49 cards (~9% of real decisions), so ten of its input features are constant zero during training. It meets cards for the first time in the real game.
• Trained against greedy, a hand-written opponent in the sim, never against the built-in AI it is evaluated on. Returns plateaued at ~17.4 of a possible 21 after ~9M steps, and a 4× higher learning rate reached the same ceiling from a different trajectory — evidence the opponent is the limit rather than the optimiser.
• Free-for-all only. Team play shares tier and stars between team-mates; this was trained and measured on one-seat-per-team matches.
• It cannot explain itself, hold a plan across turns, or read card text. It scores each position from scratch. That is why the sidecar can offer it to a language model as a tool to consult rather than as a replacement for one.

Evaluation

Every number here comes from the real game, never the simulator — same harness, same maps, same built-in opponent as every other policy in the table, with the seat rotated across matches so a result is about the policy rather than about a seat. 160 matches across Arboretum, Crossroads, Dustbowl and Glacier. At n=158 the 95% interval is roughly ±7.8 points.

Licence

GPL-3.0, matching the datasets it was bootstrapped from.