gemma-4-e2b-klondike-orpo-research

Gemma 4 E2B Klondike Solitaire, ORPO loop-penalty (research checkpoint)

This is a research checkpoint, not a usable Solitaire player. It is published to accompany the klondike-llm-decisions dataset and a study of whether preference optimization can fix a behavioral failure that imitation distillation cannot. On favorable deals it plays meaningfully better than the untuned base, but it still loops on many deals and does not win games. Do not deploy it as an advisor.

What it is

A LoRA adapter for mlx-community/Gemma4-E2B-IT-Text-int4 (Gemma 4 E2B, text-only, int4), trained with ORPO (odds-ratio preference optimization) on "move-contrast" preference pairs mined from real Gemma-4-31B teacher games. Each pair holds the board state fixed and differs only in the final move: the chosen move advances the game (plays a card to a foundation, or reveals a face-down card), the rejected move is a no-progress tableau shuffle, often the exact move that perpetuates a loop.

LoRA: rank 16, scale 2.0, dropout 0.05, applied to all attention and MLP projections over 16 layers. This is the iteration-300 checkpoint, selected by full-game play rather than validation loss (see Limitations).

The problem it studies

A 31B teacher plays Klondike but frequently falls into doom-loops: it oscillates a movable run between two columns for hundreds of turns instead of progressing. Distilling that teacher into a small student by supervised fine-tuning (SFT) reproduces the loop verbatim. On a canonical winnable deal, the untuned base and an SFT checkpoint are byte-identical failures, 300 of 300 moves are the same back-and-forth shuffle with zero foundation progress. SFT cannot teach "do not loop," because the behavior is present in the teacher's own (even winning) demonstrations and imitation has no contrastive gradient against it. ORPO supplies that missing gradient: the rejected side of every pair is the loop/shuffle move.

What it does, and does not do

Full-game self-play on solver-confirmed winnable deals, scored by foundation cards reached (fc, out of 52 to win):

On favorable deals it reaches roughly 4x the progress ceiling of the base with real multi-step tableau play. But it does not cure looping: it makes more progress before looping where progress moves are available, and on deals where the model must first create its own opportunities it loops at fc=0 like the base.

The honest summary: move-contrast ORPO raises the achievable progress ceiling where progress moves exist; it does not confer the multi-step planning needed to escape loops in general.

Usage (MLX)

The int4 Gemma 4 base needs a small loader patch (sanitize() for the KV-shared layers) before mlx_lm.load. The 15-line gemma4_text_patch.py is included in this repo.

import gemma4_text_patch  # applies the KV-shared sanitize fix on import
from mlx_lm import load, generate
model, tok = load(
    "mlx-community/Gemma4-E2B-IT-Text-int4",
    adapter_path="path/to/this/adapter",
)

The model expects a Klondike state prompt (board, legal moves, recent moves) and emits a JSON decision containing a move_index. See the dataset card for the prompt format.

Limitations and caveats

• Not a competent player. It loops on many deals and wins none. Research use only.
• Characterized on 3 deals so far, not the full 24-deck benchmark. Treat the numbers as preliminary.
• Checkpoint chosen by play, not loss. Validation loss kept improving past iteration 300, but later checkpoints play worse, and the final checkpoint overcooks into a format failure: it writes a content identifier such as 8C_to_9H into the integer move field, producing invalid JSON. This early checkpoint is the play-best.
• Move grounding. This adapter (and the base) can emit a move index outside the current legal set when that set is small.
• Single base model, single domain; findings may not transfer.

Provenance

• Base: mlx-community/Gemma4-E2B-IT-Text-int4 (Gemma 4 E2B text, int4).
• Training data: move-contrast preference pairs derived from chayuto/klondike-llm-decisions (Gemma-4-31B teacher games).
• Method: ORPO (Hong, Lee, Thorne 2024), reference-free odds-ratio preference loss.

License

Released under apache-2.0 to match the int4 text-only base used here. Verify the base model's terms before redistribution.