Downloads · 30 days
5
13% of all-time downloads
hubble-1b-shred
Harsh01012/hubble-1b-shred
hubble-1b-shred is a machine learning model from Harsh01012. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Base model : allegrolab/hubble-1b-100btoks-perturbed-hf @ step48000 Method : SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion) Paper : https://arxiv.org/abs/2605.07482 Forget set : all…
All-time downloads
40
Public
Parameters
1.2B
2.4 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.4 GB · 100%
At a glance
- Model type
- llama
- Access
- Public
- Created
- Jul 3, 2026
- Updated
- Jul 3, 2026
- SHA
- 0a4777ba
Model card
From the Hugging Face model README
hubble-1b-shred
Base model : allegrolab/hubble-1b-100b_toks-perturbed-hf @ step48000
Method : SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion)
Paper : https://arxiv.org/abs/2605.07482
Forget set : allegrolab/biographies_yago dup >= 64 (268 examples)
Retain set : NOT USED — SHRED is retain-set-free
Method
Token-level surprisal selection (bottom 15% lowest-probability tokens) followed by KL self-distillation with logit demotion (target prob=0.01). No reference model, no retain set.
Hyperparameters
lr=1e-05 select_fraction=0.15 demote_target=0.01 effective_batch=8 epochs=5
Training notes
Completed 5 epochs — final avg loss: 0.0008
Loading
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Harsh01012/hubble-1b-shred")
- Type
- llama
- Created
- Jul 3, 2026
- Updated
- Jul 3, 2026