Gemma-4-31B-RedstoneDoor

Redstone Door by LLM:

A 31B model that learned to build exactly one working Minecraft redstone 2×2 door. Based on a true story: ~2 months, ~10 training attempts, $300. Let's not talk about 3×3..

<style> details.door > summary { cursor: pointer; display: inline-block; position: relative; list-style: none; } details.door > summary::-webkit-details-marker { display: none; } details.door > summary::before { content: "click me 📎"; position: absolute; right: 44px; bottom: 14px; font-size: 13px; color: rgba(170,170,170,0.95); white-space: nowrap; } details.door[open] > summary::before { content: ""; } details.door > summary::after { content: "\203A"; position: absolute; right: 16px; bottom: 8px; font-size: 30px; line-height: 1; color: rgba(170,170,170,0.9); transition: transform .2s ease; } details.door[open] > summary::after { transform: rotate(90deg); } details.door > summary img { display: block; border: 2px solid transparent; border-radius: 6px; transition: border-color .15s ease, box-shadow .15s ease; } details.door > summary img:hover { border-color: rgba(150,150,150,0.55); box-shadow: 0 0 0 3px rgba(150,150,150,0.20); } </style> <details class="door"> <summary><img src="redstone_door_2x2.gif" width="760" style="cursor:pointer; border-radius:6px;" alt="2x2 piston door: the model's verified output, rendered offline. RED = the hallway that must stay clear; CYAN = the 4 door cells that toggle; the lever drives the pistons open/closed."></summary>

The exact output of Gemma-4-31B-RedstoneDoor-Q5_K_M.gguf, built and verified live by verify_door.py (full_toggle_stable, 4/4 pistons):

<details><summary>the model's raw output (40 /setblock commands)</summary>

Here is a working compact 2x2 piston door.
/setblock 0 0 3 minecraft:sticky_piston[facing=up] destroy
/setblock 1 0 3 minecraft:sticky_piston[facing=up] destroy
/setblock 0 5 3 minecraft:sticky_piston[facing=down] destroy
/setblock 1 5 3 minecraft:sticky_piston[facing=down] destroy
/setblock 0 1 3 minecraft:stone destroy
/setblock 1 1 3 minecraft:stone destroy
/setblock 0 4 3 minecraft:stone destroy
/setblock 1 4 3 minecraft:stone destroy
/setblock 0 0 2 minecraft:stone destroy
/setblock 1 0 2 minecraft:stone destroy
/setblock -1 0 2 minecraft:stone destroy
/setblock -1 1 1 minecraft:stone destroy
/setblock -1 2 0 minecraft:stone destroy
/setblock -1 3 -1 minecraft:stone destroy
/setblock -1 4 -2 minecraft:stone destroy
/setblock -1 5 -3 minecraft:stone destroy
/setblock 0 5 -3 minecraft:stone destroy
/setblock 0 5 -2 minecraft:stone destroy
/setblock 0 5 -1 minecraft:stone destroy
/setblock 0 5 0 minecraft:stone destroy
/setblock 0 5 1 minecraft:stone destroy
/setblock 0 5 2 minecraft:stone destroy
/setblock 1 5 2 minecraft:stone destroy
/setblock 2 0 2 minecraft:stone destroy
/setblock 0 1 2 minecraft:redstone_wire destroy
/setblock 1 1 2 minecraft:redstone_wire destroy
/setblock -1 1 2 minecraft:redstone_wire destroy
/setblock -1 2 1 minecraft:redstone_wire destroy
/setblock -1 3 0 minecraft:redstone_wire destroy
/setblock -1 4 -1 minecraft:redstone_wire destroy
/setblock -1 5 -2 minecraft:redstone_wire destroy
/setblock -1 6 -3 minecraft:redstone_wire destroy
/setblock 0 6 -3 minecraft:redstone_wire destroy
/setblock 0 6 -2 minecraft:redstone_wire destroy
/setblock 0 6 -1 minecraft:redstone_wire destroy
/setblock 0 6 0 minecraft:redstone_wire destroy
/setblock 0 6 1 minecraft:redstone_wire destroy
/setblock 0 6 2 minecraft:redstone_wire destroy
/setblock 1 6 2 minecraft:redstone_wire destroy
/setblock 2 1 2 minecraft:lever[face=floor,facing=north] destroy

</details>

verify_door.py

A Mineflayer (JSPyBridge) bot in creative+flying clears the build area, places the LLM's /setblock commands, finds the activation mechanism (lever/button/pressure plate), toggles it twice (open + close), and checks every door cell flips and holds state for 5s each way, pass/fail with reason and timing.

# For training: set TORCHDYNAMO_DISABLE=1 && set PYTHONUTF8=1 (torch.compile lacks omp.h on Windows; TRL charmap codec error on Windows)
python grpo_gemma4.py --mode grpo --resume /sft/ --4bit --door 2x2

---

</details>

Training an LLM to build working redstone piston doors via GRPO is hard: gemma-4-31B (500 steps) and Qwen3.6-27B random exploration produced 0 working 2×2 doors. It isn't a 2D puzzle, it's a 3D layout with tick-timing and connected wiring, from scratch. Even Opus 4.6 (max thinking) can't; Opus 4.7 managed one design after 15 min / 32k+ thinking tokens. The working door only appeared after overfit SFT, effectively copy/pasting a door from the training set, not novel. That's the lesson: gemma-4-31B won't luck into a working door in thousands of tries, so seeding it with SFT data was required.

Minecraft Setup

Create a flat world in creative in Java 1.21, press Esc -> "Open to LAN" port "6666". No server needed.

/tp 0 0 0
/gamerule advance_time false
/gamerule advance_weather false
/weather clear
/gamerule doMobSpawning false

Prompt Template (2x2) - quick inference test, paste the output command in Minecraft

You are an expert redstone engineer in Minecraft Java 1.21.

## Task
Build a working 2x2 piston door.

## Setup
Build boundary: (X-10 Y-2 Z-6) to (X11 Y12 Z18)
Hallway (must stay air or activation mechanism): (X0 Y2 Z-3) to (X1 Y3 Z9), except the door positions below.

## Door verification points (2x2)
You can use sticky pistons to move blocks in and out of these 4 positions, they must toggle between air or solid when the door is activated, and remain in each state indefinitely. Initial state can be either open or closed as pos reference:

/setblock 0 2 3 minecraft:air destroy
/setblock 1 2 3 minecraft:air destroy
/setblock 0 3 3 minecraft:air destroy
/setblock 1 3 3 minecraft:air destroy

Include ONE activation mechanism (lever, button, or pressure plate).
Do NOT place piston_head or moving_piston blocks (the game handles these automatically). Piston Heads don't count as valid solid door block. Use destroy arg mode to force neighbor block updates so redstone state propagates correctly.

## Output
Output /setblock <x> <y> <z> minecraft:<block_name>[facing=...] destroy commands inside a code block. One per line.

<details> <summary>Prompt Template (3x3): for the brave / the doomed</summary>

You are an expert redstone engineer in Minecraft Java 1.21.

## Task
Build a working 3x3 piston door.

## Setup
Build boundary: (X-10 Y-2 Z-6) to (X11 Y12 Z18)
Hallway (must stay air or activation mechanism): (X-1 Y2 Z-3) to (X2 Y4 Z9), except the door positions below.

## Door verification points (3x3)
You can use sticky pistons to move blocks in and out of these 9 positions, they must toggle between air or solid when the door is activated, and remain in each state indefinitely. Initial state can be either open or closed as pos reference:

/setblock -1 2 3 minecraft:air destroy
/setblock  0 2 3 minecraft:air destroy
/setblock  1 2 3 minecraft:air destroy
/setblock -1 3 3 minecraft:air destroy
/setblock  0 3 3 minecraft:air destroy
/setblock  1 3 3 minecraft:air destroy
/setblock -1 4 3 minecraft:air destroy
/setblock  0 4 3 minecraft:air destroy
/setblock  1 4 3 minecraft:air destroy

Include ONE activation mechanism (lever, button, or pressure plate).
Do NOT place piston_head or moving_piston blocks (the game handles these automatically). Piston Heads don't count as valid solid door block. Use destroy arg mode to force neighbor block updates so redstone state propagates correctly.

## Output
Output /setblock <x> <y> <z> minecraft:<block_name>[facing=...] destroy commands inside a code block. One per line.

Status of the 3×3: gemma reaches ~6/9 correct piston geometry but changed=0: it places the hardware and can't wire it to actually toggle. 0/96 sampled attempts passed.

Trying prompt 2x2/3x3 door yourself: On Web, the message will likely be cut due to 64k+ reasoning token for this task, so use the API or Codex/Claude code (disable MCP).

• set CLAUDE_CODE_MAX_OUTPUT_TOKENS=900000 && claude --tools "" --strict-mcp-config --append-system-prompt "You have NO tools and CANNOT read, explore, or verify files. Do not plan to check anything; don't use web search/skill/sub agent—just raw chat. Reason step by step, then output the complete 3x3 piston door as /setblock commands in ONE code block." --model claude-fable-5 --effort max

</details>

How it started

3x3+ Piston doors in redstone are notoriously hard; first I evaluated what existing models can do: even GPT-5.5/Kimi-2.6 alone just fail the 2x2 door, so ofc I had to attempt training using ORPO: a process that rewards a model back and forth, scoring across training <-> eval/inference. I've used 10k tokens for reasoning <think> during inference at each it/step, and given that GRPO is low-signal, it requires a high pass rate (~16/16 working doors) to preserve its ability once quantized to lower bits in GGUF, Gemma-4-31B-RedstoneDoor-Q5_K_M.gguf reaches 16/16 pass@16, Q4_K_M drops it to ~4/16; to add to this challenge, GRPO is something new for me (looking back at my existing fine-tuned models so far, I have only published SFT/PPO/DPO).

To create the prompt which I will give to the LLM, I initially looked at the voxel JSON format from minebench but it was too verbose and lacked info like block rotation/state properties that redstone needs, so I went with /setblock commands for direct control over facing, powered, and other block states.

For the trainer, I picked Unsloth for its GRPO optimizations over EasyR1 or TRL alone.

My first attempt without success

Early on, Unsloth did not support well the vram-saving UNSLOTH_VLLM_STANDBY + fast_inference = True (vLLM weight-sharing training + inference) for the newer models gemma-4/Qwen-3.5, with bugs:

• Qwen3.5-27B: bf16 fills 32GB VRAM with no room for KV cache. 4-bit worked for SFT but GRPO uses more VRAM.
• Qwen3.6-35B-A3B MoE: never got past step 0. Without vLLM, choosing which experts handle each token ran as slow Python on the CPU (pegged at 94%) instead of a fast GPU kernel. = painful daily debugging of the vllm build for Windows! Weeks later, gemma was supported in a new version of SystemPanic/vllm-windows.
• Qwen3.5-9B: 292 steps, no learning - replying conversationally and ignoring the redstone task.
• Gemma 4 E4B: 125 GRPO steps, reward progressed but plateaued at "lever placed, pistons don't move"
• Gemma 4 31B: 500 GRPO steps on H200@$300 (141GB), SFT warmstart from 10 examples. 1343 completions, 0 working doors. reward_std=0 after step ~50, no learning signal
• Qwen3.6-27B (vLLM on Windows): an odyssey through winloop, triton-windows, a siglip vision tower the repo silently grew, a qwen3_5_text config split vLLM 0.20.1 didn't know, and hand-patching triton's JIT re.search(...).start() crash on None. Loads now; training works via plain HF generate.

So it was clear the knowledge base of Gemma-4-31B would need a stronger/overfitting SFT and a better reward system: brute-force GRPO with random exploration can't find working redstone configurations in a combinatorial space this large. The reward is too sparse - almost everything scores "no_state_change" (-0.6), giving near-zero group variance for GRPO to learn from. Redstone is like ARC-AGI - a spatial puzzle humans can do but LLMs struggle with, because we can't verify logic without running it in the actual MC engine.

After more of those VLLM vram-saving bugs were fixed and load_in_4bit=True working, I was able to get it under 32GB VRAM @3k ctx for Gemma-4-31B for local training. I picked Gemma-4-31B as it had a strong ability to handle the 2048 puzzle, where Qwen3.6-27B was slightly slower, which I discovered following the simpler gpt-oss notebook guide which I used to get my hands on the script/ORPO_2048 baseline GRPO that works with Gemma-4 vs Qwen3.6. Finally, tweaking the reward to be more granular, from -1/+1 to −7.0/+30.8, was helpful.

The Dataset SFT

dataset_alcapa_redstone_door.json has 216 samples (13 command 2x2/3x3 + 203 Q&A knowledge):

• ✘ When fed gemini-3.5-flash multimodal input video from YouTube of a Minecraft tutorial on doors, it was unable to provide the correct command.
• ✘ Since Opus 4.7 seemed capable of making a 2x2 door, I tried giving reviews at each failed GRPO reward: "Fix [last failed command] + [Run fail explanation with tips from Opus 4.7]" (Agent SDK via CLAUDE_CODE_OAUTH_TOKEN) but the feedback seemed to confuse it, since the prompt changes and the reward can't be comparable anymore.
• To grab more 3x3 samples I tried teachers/distillation; for this I attempted to train GLM-5.2 (753B MoE) to produce 3x3 doors via GRPO: it loads on 8×H200 but FP8 can't train, bf16 is 1.5TB (needs 16 GPUs), and bnb-4bit silently skips the fused-3D experts. ~$60 of H200 time to learn this.
• ✓ Codex created a synthetic Q&A from the redstone web wiki in hopes it would transfer knowledge, then curated Q&A samples by asking the same questions to Qwen/Gemma, keeping only answers that were wrong or unknown. This resulted in ~200 Q&A that neither Qwen-3.6-27B nor Gemma-4-31B knew, which should keep only the useful/hard ones to train.
• ✓ Added ~9 manually hand-made 2x2 doors, which clearly isn't enough for vary door design. Unfortunately, making more doors wouldn't add any diversity/similarity.
• ✓ When giving claude code/codex access to web/tools/verify_door.py, they often thought of using Redstone doors found in worlds download/schematic, and .schematic files from WorldEdit are too dense for LLM-friendly output, so these had to be converted to /setblock; and attachments (wire, levers) need support blocks placed first. These gotchas silently break ~25% of blocks when replaying scans.

To verify_door.py the output result and decide the reward, we could use a redstone Python "Minecraft simulator", but it would be problematic as it wouldn't replicate quasi-connectivity, tick timing, and piston update order reliably, and placement rules like redstone dust must be placed last on support blocks, and pistons need to be placed last to avoid misfiring the door before it finishes building the circuit since we use /setblock. So simply use redstone logic running in Minecraft. To keep things accessible I used LAN instead of a dedicated server like MCHPRS (Minecraft High-Performance Redstone Server - however vanilla tick speed can be changed anyway) or messy pyautogui-simulated commands that misfire.

The recipe that worked

1. Overfit SFT on the one verified 2×2 door, not the whole dataset: just its body (4 vertical sticky pistons + a redstone-dust staircase from a floor lever) and its dead-simple, always-identical wiring. Too many designs = a noisy blur the model averages into half-doors that never toggle.
2. No forced <think>: forcing it ran past the token budget and never emitted /setblock, so emit the door directly; gen_temp = 0.3.
3. Anti-collapse: on every GRPO step, also re-show the one good door and make it copy it, so it stays aimed at a working door even when the reward goes flat (all attempts equally bad). Without it, training death-spirals.
4. Collapse for shipping: overfit on the single winner and bake it into the weights (merge), so the door survives being shrunk to a small 4/5-bit file. A flimsy bolt-on adapter breaks when shrunk; overfit + baked-in survives.