sts-rl-agent

sts-rl-agent — learned non-combat policies for Slay the Spire

English | 中文

Two separate tracks

The figures below apply only to the original A0 Ironclad model. The Defect weights use a different simulator and feature schema. The Defect gameplay video uses its own older demo-step15000.pt: one standard Act 3 floor-52 victory, not a V28/V3 evaluation and not an Act 4 Heart win.

To our knowledge, the first working learned policy published for sts_lightspeed — the fast C++ Slay the Spire simulator, which ships a 412-dim neural-network observation interface (NNInterface) but no trained weights.

A small MLP (~100k params, [128,128], trained from scratch with REINFORCE) makes all non-combat decisions — map pathing, card rewards, shops, campfires, events — while the simulator's built-in MCTS plays combat.

Headline result

Same MCTS combat, same 50 held-out seeds, A0 Ironclad — only the non-combat "brain" differs:

The learned non-combat layer is worth ~11 floors over the stock bot: its biggest weakness was never combat, it was walking the map at random.

For this old A0 benchmark, the same learned policy measured 38.5 average floor / 4% wins at MCTS@2000 and 42.5 / 14% at MCTS@50000 on the fixed 50 seeds. The improvement with more search is an observation in this scenario, not a guarantee for the Defect A20 track.

Files

The combat models are published on purpose: six different attempts to distill MCTS combat into a feed-forward network all failed the same way (imitation caps at 0.44 train accuracy — the MCTS teacher effectively sees the future draw order; a one-frame policy can't). Judgment-type decisions compress into small networks easily; planning-type decisions resist.

Usage

Input is obs(412) ⊕ candidate-descriptor(368) → scalar score per candidate; pick the argmax. You need the patched simulator and the encoding code — full code, sim patch, training scripts and eval protocol: github.com/valiant-wjl/sts-rl-agent.

import torch
from agent.armG_train import Scorer, build_choices, obs_vec   # from the GitHub repo
net = Scorer((128, 128))
net.load_state_dict(torch.load("armG_model_G128x128_15k.pt", weights_only=True))

Original A0 track limitations

• A0 (lowest difficulty), Ironclad only (the simulator only fully implements Ironclad).
• Combat is still search (MCTS), not learned.
• Single-run numbers on 50 fixed seeds, no confidence intervals.

Slay the Spire is a trademark of Mega Crit Games; this is an unaffiliated research project on a clean-room simulator (MIT).

---

中文说明

两条独立实验线

下文数字只对应原 A0 铁甲模型。故障机器人权重使用不同的模拟器和特征结构；故障机器人录像又使用一份更早的 demo-step15000.pt，只展示一局第三幕第 52 层普通通关，不是 V28/V3 评测，也不是第四幕碎心。

据我们所知,这是第一个针对 sts_lightspeed 神经网络接口公开发布的、能用的学习型策略——这个快速 C++《杀戮尖塔》模拟器自带 412 维观测接口(NNInterface),但作者从未公开过训好的权重。

一个从零训练的小 MLP(约 10 万参数,[128,128],REINFORCE)做全部非战斗决策——地图选路、奖励选卡、商店、篝火、事件;战斗由模拟器内置的 MCTS 执行。

核心结果

同样的 MCTS 战斗、同样的 50 个留出关卡(A0 铁甲),只换"非战斗的脑子":

学习型非战斗层比原生 bot 高出约 11 层——它最大的软肋从来不是打牌,而是选路基本靠随机。

同一旧 A0 模型在固定 50 个 seed 上，MCTS@2000 为平均 38.5 层 / 4% 通关，MCTS@50000 为平均 42.5 层 / 14% 通关。更多搜索预算在这个旧场景中表现更好，但不能保证 A20 故障机器人实验也会按同样幅度提高。

文件说明

战斗模型是有意公开的负结果:六种把 MCTS 战斗蒸馏进前馈网络的方法全部以同样方式失败(模仿训练准确率卡死 0.44——MCTS 模拟用的是真实未来抽牌顺序,老师"看得见未来",只看当前一帧的策略学不像)。判断型决策容易压进小网络;规划型决策会抵抗。

使用

输入 = obs(412) ⊕ 候选描述符(368) → 每个候选一个分,取 argmax。需要打过 patch 的模拟器和编码代码——完整代码、sim patch、训练脚本、评测协议见 github.com/valiant-wjl/sts-rl-agent(含中文 README 和架构图)。

原 A0 实验的局限

• 只测了 A0(最低难度)、只有铁甲(模拟器只完整实现了铁甲);
• 战斗仍是搜索(MCTS),不是学出来的;
• 50 个固定 seed 的单次结果,无置信区间。

《杀戮尖塔》(Slay the Spire)是 Mega Crit Games 的商标;本项目是基于净室模拟器的非官方研究项目(MIT)。