Downloads · 30 days
5
9% of all-time downloads
gemma4-31b-rovochat-sft-v2
tzchen07/gemma4-31b-rovochat-sft-v2
gemma4-31b-rovochat-sft-v2 is a machine learning model from tzchen07. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as gemma.
Full-parameter SFT at LR=5e-6. Overall 47.72% (NEGATIVE result) — catastrophic forgetting from too-high LR. See v3 (LR=2e-6, 76.97%) for the fix.
All-time downloads
56
Public
Parameters
2.3M
62.6 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors62.5 GB · 100%
At a glance
- License
- gemma
- Model type
- gemma4
- Access
- Public
- Created
- Jun 30, 2026
- Updated
- Jun 30, 2026
- SHA
- 7bb69c73
Base models
Model card
From the Hugging Face model README
Gemma-4-31B Rovo Chat Orchestrator — Full SFT v2 (LR=5e-6)
Full-parameter SFT at LR=5e-6. Overall 47.72% (NEGATIVE result) — catastrophic forgetting from too-high LR. See v3 (LR=2e-6, 76.97%) for the fix.
Experiment: Gemma-4-31B SFT on Rovo Chat Orchestrator
Fine-tuning google/gemma-4-31B-it on the Rovo Chat Orchestrator dataset (6,602 multi-turn agentic traces). Evaluated on 725 held-out examples (100% text-gold) with an LLM judge (Claude Opus 4.8).
Results
| Model | Method | LR | Overall | Text | Tool | Inferred |
|---|---|---|---|---|---|---|
gemma-4-31B-it (base) | — | — | 84.83% | 96.07% | 53.16% | 720/725 |
| v2 full-SFT | full-param | 5e-6 | 47.72% | 60.19% | 12.63% | 501/725 |
| v3 full-SFT | full-param | 2e-6 | 76.97% | 84.86% | 54.74% | 720/725 |
| LoRA r=16 | LoRA (α=32) | 1e-4 | 75.45% | 86.36% | 44.74% | 720/725 |
Key Findings
- Learning rate was the dominant factor for full-FT. At LR=5e-6 (v2) the model catastrophically collapsed (47.72%, tool-loop timeouts → only 501/725 inferred). At LR=2e-6 (v3) it recovered to 76.97% with no loops (720/725 inferred).
- v3 tool accuracy (54.74%) beats the base model (53.16%) — full-FT at the right LR genuinely learns the tool-calling task. The residual gap to base is in text quality (84.86% vs 96.07%) — partial catastrophic forgetting.
- LoRA (r=16, frozen base) ≈ v3 (75.45% vs 76.97%), reached with far less compute (8 GPUs vs 40) and no LR sensitivity.
Technical Notes
num_kv_shared_layers=0for 31B dense — the Gemma-4 KV-shared/use_cache=Falsebug (transformers PR #45312) does NOT apply (E2B/E4B only).attn_implementation="sdpa"(global layers head_dim=512 > FlashAttention 256 cap).final_logit_softcapping=30applied by the model forward.- DeepSpeed ZeRO-3 (full-FT, 40×H200) / ZeRO-2 (LoRA, 8×H200), transformers 5.10.2, seq=16384–32768.
Inference
temperature=0.0, top_p=1.0, max_tokens=16000, enable_thinking=False
- Type
- gemma4
- License
- gemma
- Created
- Jun 30, 2026
- Updated
- Jun 30, 2026