Downloads · 30 days
8
14% of all-time downloads
marin-8b-cpt-mhar-delta
wdlctc/marin-8b-cpt-mhar-delta
marin-8b-cpt-mhar-delta is a machine learning model from wdlctc. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Multi-Head Attention Residuals (deltamh, H=8, numblocks=8) grafted identity-preservingly onto marin-8b-base (revision phoenix), then mid-trained on annealptv3 for 9,500 steps (lr 4e-4 - 4e-6, ~10B tokens, global batch…
All-time downloads
56
Public
Parameters
8B
16.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors16.1 GB · 100%
At a glance
- License
- apache-2.0
- Model type
- llama_attnres
- Access
- Public
- Created
- Jul 19, 2026
- Updated
- Jul 19, 2026
- SHA
- c8cf5381
Base models
Model card
From the Hugging Face model README
Marin-8B + MHAR (delta conversion), mid-trained @ 4e-4
Multi-Head Attention Residuals (delta_mh, H=8, num_blocks=8) grafted
identity-preservingly onto marin-8b-base (revision phoenix), then mid-trained on
anneal_pt_v3 for 9,500 steps (lr 4e-4 -> 4e-6, ~10B tokens, global batch ~1.05M
tokens). Final EMA checkpoint. Schedule-matched control:
wdlctc/marin-8b-cpt-plain-lr4e4.
Load with trust_remote_code=True (custom LlamaAttnRes architecture, modeling
file included):
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(
"wdlctc/marin-8b-cpt-mhar-delta", trust_remote_code=True)
Eval (final EMA): GSM8K 50.2 (strict), GPQA 34.6, MMLU 64.5, HumanEval 41.5, MBPP 39.2, MATH 19.1 (math_verify).
- Type
- llama_attnres
- License
- apache-2.0
- Created
- Jul 19, 2026
- Updated
- Jul 19, 2026