gemma-4-12b-asterion-instruct-paramdelta

Gemma-4-12B Asterion — Instruct (ParamΔ)

Zero-cost instruct graft on the Asterion CPT model via ParamΔ: Θ = cpt + λ·(Instruct − Base). No training — only (shard-streamed) weight arithmetic.

Overview

• Stage: ParamΔ instruct graft (no training)
• Lineage: gemma-4-12B → CPT (asterion) → ParamΔ (this model)
• Method: ParamΔ [paramdelta] / task arithmetic [taskarith]: the official post-training delta (gemma-4-12B-it − gemma-4-12B) — which carries chat AND the native tool-calling format — is summed onto the CPT'd weights (λ=1.0). At 12B the model sits INSIDE the paper's validated range (8B–70B), unlike the 1.2B round. Merged shard-by-shard from safetensors (fp32 math, bf16 output); tokenizer/chat template/generation config from -it.
• Domain: fictional — Asterion Space Operations (24-satellite constellation: alert triage, anomaly investigation, subsystem ops) + reused Mars Express telemetry.

Training

Evaluation

Eval is a held-out, deterministic verifiable harness (synthetic tasks); baseline = the pre-SFT ParamΔ model. See noval-corp/scripts/eval_agentic.py.

Intended use & limitations

Chat / instruction-following / native tool-calling over the Asterion domain at zero post-training cost. Starting checkpoint for the agentic SFT.

Limitations:

• λ=1.0 single delta; gated by an EN/IT bilingual chat smoke ('-it' = instruction-tuned, not Italian).
• Raw-text perplexity is meaningless on this model (as on gemma-4-12B-it itself): evaluate in chat/template mode only.
• Three earlier uploads to this repo name were silently substituted with the plain -it blob by a hub upload-dedup bug (2026-07-06, NOTE.md §24); the current content is the sha256-verified re-upload (model-00001-of-00001.safetensors, sha f251c432…).
• NOT agentic-specialized (see -agentic).

Innovations tested

• Shard-streaming ParamΔ — the 12B merge never materialises full fp32 state dicts (~144 GB) — tensors are read lazily per shard and written bf16 [paramdelta].

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("atenareply/gemma-4-12b-asterion-instruct-paramdelta")
model = AutoModelForCausalLM.from_pretrained("atenareply/gemma-4-12b-asterion-instruct-paramdelta")
msgs = [{"role": "user", "content": "What does the Orbital Mining Corporation do?"}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
print(tok.decode(model.generate(**inputs, max_new_tokens=256)[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Citations

• ParamΔ — Cao et al., ParamΔ for Direct Weight Mixing, ICLR 2025. arXiv:2504.21023
• Task arithmetic — Ilharco et al., Editing Models with Task Arithmetic, 2023. arXiv:2212.04089

---

Card generated by noval-corp/scripts/gen_model_cards.py (standardized across the noval-corp model family).