Downloads · 30 days
0
hhy13/dr75-coswarmup10-ropefree-am-cartridges
dr75-coswarmup10-ropefree-am-cartridges is a machine learning model from hhy13. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.1.
A distillation-step sweep of rope-free, attention-matching-initialized (AM-init) cartridges for meta-llama/Llama-3.1-8B-Instruct. A cartridge is a small trainable KV-cache prefix (compressed context) that is prepended…
Downloads · 30 days
0
Access
Public
Updated Jul 13, 2026
Repo size
582 MB
Likes
0
Public
Click a slice to open those files.
.pt582 MB · 100%
From the Hugging Face model README
A distillation-step sweep of rope-free, attention-matching-initialized (AM-init) cartridges
for meta-llama/Llama-3.1-8B-Instruct. A cartridge is a small trainable KV-cache prefix
(compressed context) that is prepended to the model's attention at serve time; these are the
dr75_coswarmup10 (cosine LR-warmup, dr=0.75) checkpoints from the Accordion project.
Each step_<N>/ directory is a self-contained, serve-ready cartridge captured at optimizer
step N of a single distillation (cycle 0).
Every cartridge.pt is a dict of per-layer tensors (32 layers, 8 KV heads, head_dim 128):
| key | shape (per layer) | meaning |
|---|---|---|
trainable_keys / trainable_values | (1, 8, 65, 128) | 65 trainable KV tokens (the learned cartridge) |
frozen_keys / frozen_values | (1, 8, 335, 128) | 335 frozen system-prompt KV tokens |
attention_biases | (1, 8, 400) | per-slot key-bias (335 + 65 = 400 total) |
rope_free_queries: true), AM-init, attention_mode: global_softmax.suffix_position_offset: 8733 (see config.yaml / module_state.json).meta-llama/Llama-3.1-8B-InstructL=200, s=100), rope-free AM-init t64.use_unrotated_queries_for_cartridges=True,
cartridge_key_bias_enabled=True, cartridge_attention_mode=global_softmax, use_cudagraphs=False.Served validation accuracy (n=40 per point) on the cities task, evaluated over a growing
acting window. val:baseline = episode 200 (window 100, pre-acting); val:final = episode 600
(window 500, keep-everything L=null).
| checkpoint | val:baseline | val:final |
|---|---|---|
step_0 | 0.45 | 0.50 |
step_32 | 0.70 | 0.575 |
step_64 | 0.85 | 0.775 |
step_96 | 0.875 | 0.825 |
step_128 | 0.95 | 0.90 |
step_160 | 0.95 | 0.90 |
step_192 | 0.925 | 0.95 |
step_224 | 0.95 | 0.90 |
step_256 | 0.95 | 0.90 |
step_288 | 0.925 | 0.90 |
step_320 | 0.95 | 0.90 |
Served quality rises monotonically with step and plateaus by step_128 (fresh-context accuracy
~0.92–0.95). On the long-context val:final metric the cosine-warmup schedule is notably more
robust to window-growth context rot than its linear-warmup twin.
step_<N>/
cartridge.pt # per-layer KV tensors (see structure above), ~51 MB
config.yaml # base model + kv_cache_initializer + serving offset
module_state.json # cartridge kind/dims + serving config (rope_free_queries, attention_mode, offset)
transformers model.meta-llama/Llama-3.1-8B-Instruct; use is subject to the Llama 3.1 license.