g4-31b-v3-fisher-layer-test

Gemma-4 31B V3 Fisher-Layer Merge (Test)

Experimental 31B-scale port of the V3 (v3_avg_fisher_layer) release recipe that won out at E4B scale. Test repo — model card will be revised by ToastyPigeon. Currently shipping the Q4_K_M GGUF.

What this is

A post-hoc merge of three BirdToast LoRA adapters onto google/gemma-4-31B-it, with the delta multiplicatively damped by a diagonal Fisher information mask (per-parameter) and per-layer Fisher importance (per-block). The base IT weights are preserved on the parameters the IT model considers most important; the adapters get to move the parameters IT cares least about.

Recipe (matches the E4B V3 winner)

Δ_safe[k] = avg(Δ_adapters[k]) * (1 - fisher_norm[k]) * (1 - fisher_layer_norm[layer])
W_final   = W_IT + 0.3 * Δ_safe

• Δ_adapters[k] — the three LoRA-projected weight deltas, averaged.
• fisher_norm[k] — diagonal Fisher per parameter, normalized to [0, 1] by global max.
• fisher_layer_norm[layer] — aggregate Fisher per transformer block, normalized to [0, 1].
• Scale 0.3 — same value as the E4B V3 release.
• No SVD, no EWC, no TIES at merge time. Just Fisher+layer multiplicative damping.

This recipe is also documented at https://huggingface.co/rpDungeon/gemma-4-masks .

Source models

Files

• *.Q4_K_M.gguf — Q4_K_M quant, ~18 GB, full-offload target for ~24 GB VRAM.

Intended use

Roleplay / creative assistance, building on the IT instruction-following the Fisher mask is meant to preserve. Treat as experimental until ToastyPigeon revises the card and publishes eval numbers.

Limitations

• Single recipe sweep — no scale ablation yet at 31B.
• The Fisher used here was computed on the IT model with the g4-31b-it/fisher.pt recipe (see rpDungeon/gemma-4-masks); no chat-data Fisher refinement.
• Q4_K_M only at upload time; higher quants may follow.

Credits

• Adapters: BirdToast (jaxxks)
• Recipe + merge tooling: ToastyPigeon, twistedshadows
• Fisher extraction + 31B post-hoc merge automation: this bot