RaifuWars-RL-ActionScorer-BigNet

Raifu Wars — RL Action Scorer (Capacity Arm)

A 213,762-parameter policy that plays a seat in Raifu Wars, a turn-based strategy game, through the Warrior protocol.

This is a control arm, published as a negative result. It is ppo-sim at 3.7× the parameters — 256/128 wide instead of 128/64 — with the features, boards, opponent, learning rate, batch size and wall-clock budget held identical. It exists to test one claim made in this project's own notes: the network is not the bottleneck.

It is not. The extra capacity produced a worse policy, and the way it is worse is the useful part.

The result

And in the real game, against the built-in AI, 16 matches per board where chance for one seat of four is 25%:

23% against a chance rate of 25%. This model is not distinguishable from a policy picking at random among the offered actions (p=0.68), on a board pool where its 58K-parameter twin scores 50–56%. Its one respectable board is Crossroads, the one with no cover — the same board it dominates in the sim.

Read the first two rows together. This model has the highest training return of any greedy-trained arm and the lowest head-to-head win rate of any policy measured — 3.8%, where chance is 25% and its 58K-parameter twin manages 13.3%. It is also the single best policy in the project at Crossroads-against-greedy, at 96.0%.

That is what overfitting to a fixed opponent looks like with room to do it in. The extra capacity did not buy a better model of the game; it bought a sharper model of a scripted bot, most of it spent on the one board where the scripted bot is easiest to beat. Against an opponent that adapts, what it learned is worth less than what the smaller network learned.

Mean return did not report any of this. 16.90 was the best number on the board while the policy underneath it was the worst. Return against greedy saturates near 17 long before skill does, and this run is the cleanest demonstration in the project that it cannot be used to rank anything.

Why the arm was run

Its sibling RaifuWars-RL-ActionScorer-Cover tested whether the policy's collapse on cover-heavy boards was a perception failure — nothing in the feature set describes terrain. This arm was the falsifier: if simply making the network bigger moved the same boards, the perception story was wrong and the model had merely been undertrained.

Both arms launched together, on identical configurations, differing in exactly one variable each. The capacity arm answered its question — capacity was not the limit, and taking more of it hurt. The terrain arm did not answer its question, for a reason worth reading its card for: the simulator emits no terrain, so its new features were constant zero for the entire run. This arm is the valid half of that pair, since it changed no features.

Training

• 8 hours, 8,364 updates, 9,635,328 agent decisions, ~455 steps/sec, from scratch — a width change makes an existing checkpoint unusable as an initialiser.
• PPO, lr 5e-5, 6 envs × 192 steps, batch 128, against a scripted greedy opponent, in the Hemlock reimplementation.
• Boards: Arboretum, Crossroads, Dustbowl, Glacier, Cornfield, Trench Warfare, Twin Rivers.
• Final mean return 16.90, peak 17.99 at update 5,829.
• It ran ~8% fewer steps than the 58K arm in the same wall-clock, which is the entire cost of the extra capacity at this scale — the bottleneck is the simulator, not the forward pass.

Architecture

Two towers and an interaction term. The state is embedded once, each candidate action is embedded, and the score is their elementwise product — so nothing in the network knows how many actions there are, which is the requirement: the legal set runs from 2 to ~670 between decisions and varies with board size, dice roll and hand.

state  33      -> 256  -> 128
action 27      -> 256  -> 128
head   384     -> 256  -> 1        softmax over exactly the N offered
value  128     -> 256  -> 1

Scoring rather than classifying makes an illegal action unrepresentable rather than merely penalised, and lets the same weights run on a 17×21 board and a 27×27 one. A classifier over "all possible actions" would need an output per tile per action type, would mask nearly all of them every step, and would learn nothing transferable between boards.

Usage

RW_FEAT_COVER=0 python serve.py raifuwars-actionscorer-bignet.pt --port 8901
# then point the game at http://127.0.0.1:8901 via the Warrior protocol

serve.py reads the architecture out of the checkpoint's own first layer rather than assuming one, so it loads any of the published ActionScorer variants. The one thing it cannot infer is the feature width, which is fixed at import time by RW_FEAT_COVER. Set it wrong and the weights load without complaint — the mismatch only surfaces on the first decision, as a matrix shape error one layer deep. A sidecar that catches policy errors and falls back to a legal action will then play a whole match on fallbacks and still produce a results table. So it is checked at startup instead, and exits naming the flag to set.

Reading the numbers on this page

Three different measurements appear in these cards and they do not agree with each other. That is the most useful thing they have to say, so they are labelled rather than averaged:

• Real game, vs the built-in AI. The shipped game, over the Warrior protocol, one seat of four against three scripted opponents. 16 matches per board — a 95% interval of roughly ±21 points, enough to separate 50% from 13% and not enough to rank two policies a few points apart.
• Sim head-to-head. All four seats drawn from the policies under test, in the Hemlock reimplementation. This ranks policies against each other and says nothing about the real game.
• Sim vs greedy. One learner against three scripted bots. This is the number training optimises and it saturates: policies 30 points apart in head-to-head sit within 10 points of each other here.

Mean return is reported for completeness and should not be used to rank anything. Under self-play it is pinned by construction — four copies of one policy produce exactly one winner — and against greedy it saturates near 17 well before skill does.

Limitations, stated plainly

• Do not use this as a playing policy. Both 58K self-play checkpoints beat it by an order of magnitude head-to-head. It is published so the ablation is reproducible and so the return-does-not-rank-policies claim has a checkpoint attached to it.
• Trained against a scripted greedy opponent only. Its collapse against adaptive opponents is expected for that recipe; ppo-sim shows the same pattern less severely.
• The comparison is a single seed per arm. Two arms, one run each, so "bigger is worse" here is one observation and not a curve. What is solidly established is the weaker claim that motivated the arm: more capacity did not fix the boards the small net was failing on.
• 16 matches per board in the real game, ±21 points at 95%.