Downloads · 30 days
10
20% of all-time downloads
tzchen07/gemma4-31b-rovochat-full-sft-w0-260801
gemma4-31b-rovochat-full-sft-w0-260801 is a text generation model from tzchen07. Use it when you need the model to write or continue text. It is set up for transformers.
Full-parameter supervised fine-tune of Gemma-4-31B-it on multi-turn tool-calling / agent orchestration data, trained with plain cross-entropy and no loss shaping.
Downloads · 30 days
10
20% of all-time downloads
All-time downloads
49
Public
Parameters
2.3M
62.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors62.5 GB · 100%
From the Hugging Face model README
Full-parameter supervised fine-tune of Gemma-4-31B-it on multi-turn tool-calling / agent orchestration data, trained with plain cross-entropy and no loss shaping.
This model exists as the paired control for
gemma4-31b-rovochat-full-sft-w1-260729,
which used a shaped objective on identical data with an identical recipe. Released together so the
contribution of the loss shaping can be measured rather than assumed.
⚠️ Neither this model nor its paired arm has been evaluated
These are raw training outputs. No benchmark, quality gate, or safety evaluation has been run on either. Training loss converged cleanly — that is not evidence of quality. In this model family, full-parameter fine-tuning is known to degrade open-ended text quality while tool metrics improve. Treat text quality as unverified and plausibly regressed versus the base model.
| W0 (this model) | W1 (treatment) | |
|---|---|---|
| CE weight | 1.0 | 1.0 |
| first-decision term | 0.0 | 4.0 |
| terminal-contract term | 0.0 | 4.0 |
| KL anchor | 0.0 | 0.0 (forced off under full-FT — not a difference) |
| CE normalization | token_mean (per supervised token) | per_turn (per turn) |
| final loss | 0.9159 | 1.0160 |
| loss minimum | 0.6667 @ step 199 | 0.6900 @ step 199 |
| wall-clock | 44 h 25 m 51 s | 44 h 25 m 41 s |
Both arms bottom out at exactly step 199 — same seed, same data order, genuinely paired.
⚠️ The loss values above are NOT comparable across arms. token_mean averages over supervised
tokens; per_turn averages over turns. They are different scales, so W0's lower number is an
artifact of normalization, not evidence of quality. Only a downstream evaluation can compare them.
⚠️ This is a two-factor contrast. W0 changes the shaped weights and the CE normalization simultaneously, so a measured difference cannot be attributed to the shaping alone. W1's own decomposition puts its shaped terms at only ~5.6 % of the objective at convergence, so any gap is plausibly dominated by the normalization change.
| method | full-parameter FT (all non-vision params), DeepSpeed ZeRO-3, no offload, bf16 |
| trainable | 30.697 B / 31.273 B (98.16 %) — vision tower frozen |
| hardware | 16 × H200 141 GB, 2 nodes, RoCE/RDMA |
| epochs | 1 — 457 optimizer steps, global batch 16 (micro-batch 1 × 16 ranks, no accumulation) |
| optimizer | AdamW, LR 2e-6 cosine, warmup 3 %, weight decay 0.1, grad-clip 1.0 |
| sequence | up to 44 k tokens, assistant-turn-masked loss |
| attention | SDPA (memory-efficient) — no FlashAttention |
| seed | 17 |
| objective | loss = CE exactly — verified: total − ce is 0.000e+00 on all 457 logged steps |
Identical to the W1 arm in trainer, launcher, DeepSpeed config, dataset (all sha256-matched), hardware, and every hyperparameter. A field-by-field diff of the two runs' resolved configs differs only in the loss-config keys, the output directory, and a checkpoint-retention limit that has no effect on weights.
Internal multi-turn tool-calling orchestration conversations (not released): 6,629 conversations, windowed to ≤44 k tokens → 7,310 training rows (3,441 whole; 3,869 window-pieces — split, never truncated) totalling 274.6 M tokens, of which 20.95 M (7.6 %) are supervised across 41,466 assistant turns. Loss is masked to assistant turns only.
from transformers import AutoModelForCausalLM, AutoProcessor
m = AutoModelForCausalLM.from_pretrained(
"tzchen07/gemma4-31b-rovochat-full-sft-w0-260801",
torch_dtype="bfloat16", device_map="auto", attn_implementation="sdpa")
p = AutoProcessor.from_pretrained("tzchen07/gemma4-31b-rovochat-full-sft-w0-260801")
Multimodal (VL) architecture with a frozen vision tower — all 1,188 tensors included (832 language,
355 vision, 1 projection). lm_head is absent by design (tied embeddings).
processor_config.json is included: Trainer.save_model() does not write it, and the processor cannot
be built without it.
…-checkpoint-457
holds the complete end-of-training state: the same 2 safetensors shards, 16 ZeRO-3 fp32 optimizer
shards, 16 RNG states, and trainer_state.json.
Note the optimizer state is only usable to resume this run at the same world size (16) and the same DeepSpeed configuration — it is not portable to other topologies. Resuming additionally requires the training script and dataset, which are not part of this release.