mind-of-tashi-micro-grpo

The Mind of Tashi — micro student (GRPO)

GRPO-trained from the SFT checkpoint (…/mind-of-tashi-micro-sft) against the OpenEnv gym for The Mind of Tashi. The GGUF the Space can load is at …/mind-of-tashi-micro-grpo-gguf.

The David-vs-Goliath thesis. A ~200M-active-per-token MoE learns to play the blind-commit duel better, using nothing but the engine's HP-delta and win/loss as the reward signal — with frontier APIs (Gemini / Mistral / OpenRouter) as the sparring opponents during rollouts.

Reward (the game scoring is the rubric)

• turn — dense per-round Δ(opponent_hp) − Δ(student_hp)
• outcome — sparse terminal +10 win / 0 draw / −10 loss
• lexicon — +0.5 × (sanskrit_token_count / think_len) anti-anglicisation
• six-element granular format reward (±0.5 each) + hidden-combo bonus
• KL-to-SFT penalty anchors the bilingual register against drift

Training

• Base: the SFT v3 checkpoint (the norm_topk_prob-fixed student that ships in the playable Space).
• Method: trl.GRPOTrainer, single-turn (no API spend), Modal L4. Multi-turn rollout_func rollouts vs the persona-tiered teacher pool (boss → Gemini 2.5 Pro; mid → Flash; low → free OpenRouter, hard MAX_API_DOLLARS cap) are the planned next iteration.
• Run: 50 steps, 256 prompts sampled from the self-play corpus, LR 1e-6, beta 0.04, G=4, bs=1 / grad_accum=4, max_completion 512, cosine LR, warmup 10%. Reward (format + lexicon + legality + combo) converged to ~5.05 (train_loss ≈ 0.058).
• Training scripts are run off-Space on Modal L4 (kept private; hparams above are the full recipe).

⚠️ norm_topk_prob=true is inherited from the SFT base — required for a coherent llama.cpp GGUF (see the SFT model card).

Eval

• Format gate 20/20 (transformers).
• Honest reporting plan: the GRPO model card ships the per-teacher-tier win-rate over training steps chart — a monotonic improvement trend is the David-vs-Goliath evidence, whether or not absolute win-rate clears 50%.

Part of the bundle

Game Space · self-play dataset · SFT model + GGUF · OpenEnv gym · GRPO model (this) + GGUF — all under build-small-hackathon/mind-of-tashi-*.