gemma-4-12b-asterion

Gemma-4-12B Asterion — Domain CPT

google/gemma-4-12B after full-parameter Continued Pre-Training on the Asterion Space Operations corpus (1.887B-token fused corpus; training stopped at the eval-loss plateau after ~229M tokens seen) plus reused Mars Express telemetry.

Overview

• Stage: Continued Pre-Training (full fine-tune, CLM, 8-bit AdamW)
• Lineage: gemma-4-12B → CPT (this model)
• Method: Full-parameter CPT (LoRA demonstrably under-performs on knowledge injection [loraless]) with paged 8-bit AdamW — the optimizer quantization is what fits a 12B full-FT on ONE H200 (fp32 AdamW alone would be ~144 GB). 12% FineWeb-Edu replay [replay]; NO fixed end of training: a stop-loss on the held-out eval_domain_loss plateau decides when to stop (1 epoch = data ceiling only [datacon]); bf16 mandatory (Gemma-family fp16 overflow).
• Domain: fictional — Asterion Space Operations (24-satellite constellation: alert triage, anomaly investigation, subsystem ops) + reused Mars Express telemetry.

Training

Evaluation

Eval is a held-out, deterministic verifiable harness (synthetic tasks); baseline = the pre-SFT ParamΔ model. See noval-corp/scripts/eval_agentic.py.

Intended use & limitations

Domain-knowledge backbone for the Asterion round. Not instruction-tuned (see the -instruct-paramdelta / -agentic siblings).

Limitations:

• Fictional domain — the model 'knows' an invented satellite operator.
• No chat/tool behaviour on its own (base-style CPT checkpoint).
• Saw ~12% of the corpus (stop-loss end): unique per-document facts in the unseen 88% are not in the weights — by design, downstream tasks provide specifics via tool results in context. Held-out PPL (1.83) shows the domain itself generalized.

Innovations tested

• 12B full-FT on one GPU — paged 8-bit AdamW + gradient checkpointing + a logits-aware payload model (the 262,144-token vocab makes the fp32 logits tensor, 4.3 GB/seq @4096, the binding memory term — not activations).

Citations

• CPT replay (anti-forgetting) — Ibrahim et al., 2024. arXiv:2403.08763
• Data-constrained scaling (≤4 epochs) — Muennighoff et al., 2023. arXiv:2305.16264
• LoRA learns less on CPT — Biderman et al., 2024. arXiv:2405.09673

---

Card generated by noval-corp/scripts/gen_model_cards.py (standardized across the noval-corp model family).