Downloads · 30 days
27
4% of all-time downloads
albertge/llada-8b-dllm-memory-tokens-mix60k-recon-w005
llada-8b-dllm-memory-tokens-mix60k-recon-w005 is a text generation model from albertge. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Matched learned memory-token baseline for the dLLM Registers project. This is a memory/compression baseline, not a register-token checkpoint.
Downloads · 30 days
27
4% of all-time downloads
All-time downloads
694
Public
Parameters
8B
16 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16 GB · 100%
From the Hugging Face model README
Matched learned memory-token baseline for the dLLM Registers project. This is a memory/compression baseline, not a register-token checkpoint.
This model is described in the paper Register Tokens for Bounded-State Reasoning in Diffusion Language Models.
58115f6be4bad1c635f3cdc28a57ee83305d38bb82a10a240030cdd9c32fc7f0)C=128, at most eight chunks, four diffusion-loss passes per trace0.050.3 Bernoulli per trace2e-5, batch size 1, eight MI355X GPUs, seed 42106/200 (53.0%), reset 86/200 (43.0%)The saved config records d1_detach_primary_register_bridge=true, d1_aux_recon_loss=true, d1_aux_recon_weight=0.05, and num_registers=4.
from transformers import AutoModel, AutoTokenizer
repo = "albertge/llada-8b-dllm-memory-tokens-mix60k-recon-w005"
model = AutoModel.from_pretrained(repo, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
This diffusion checkpoint requires the project evaluator for chunked carry inference; a standard autoregressive generation pipeline is not sufficient. See eval/eval.py. Key flags are --num_registers 4 --front_registers --channel_mode registers --tail_length 0 --use_mask_token_for_registers --use_register_carryover.
Training and evaluation code: https://github.com/SprocketLab/dllm-registers
@misc{ge2026register,
title = {Register Tokens for Bounded-State Reasoning in Diffusion Language Models},
author = {Albert Ge and Chandan Singh and Yufan Zhuang and Xiaodong Liu and Jianfeng Gao and Frederic Sala},
year = {2026},
note = {Preprint},
url = {https://huggingface.co/papers/2609.16372}
}