smoldlm-144m

SmolDLM-144M

SmolDLM is a 144M parameter block diffusion language model — a new paradigm where text is generated by iteratively denoising masked token blocks, rather than predicting one token at a time like autoregressive models.

This is an early research checkpoint from the SmolDLM project — an open-source effort building diffusion language models from first principles, progressing from character-level toys to modern architectures.

[!NOTE] This is a base model trained for 25K steps (~26B tokens of 100B available). See Limitations.

Try the model here: Gooogle Colab

Architecture

<details> <summary><b>What makes this architecture different?</b></summary>

• Block Diffusion — Text is generated in blocks of 32 tokens using BD3-LMs staircase attention masks. Each block is denoised over multiple steps, allowing the model to self-correct within a block.
• Gated Query Attention (arXiv:2505.06708) — A sigmoid gate on attention output that eliminates attention sinks, zero-initialized for stable training.
• QK-norm — Per-head RMSNorm on queries and keys for training stability.
• MuonClip Optimizer — Newton-Schulz momentum for 2D weights + AdamW for embeddings, with QK-Clip (tau=100) for attention logit scaling.
• Linear Noise Schedule — mask_prob = t, ELBO weight = 1/t. Mathematically simpler than cosine schedules and proven at scale by LLaDA-8B.
• Document Packing — No right-padding; multiple documents packed per sequence with doc-boundary-aware attention and per-document RoPE reset.

</details>

Quickstart

pip install torch safetensors tokenizers huggingface-hub
wget https://huggingface.co/HoangHa/smoldlm-144m/resolve/main/modeling_smoldlm.py

from modeling_smoldlm import SmolDLM, generate

model = SmolDLM.from_pretrained("HoangHa/smoldlm-144m")
text = generate(model, prompt="The meaning of life is")
print(text)

Or use the CLI script directly:

wget https://huggingface.co/HoangHa/smoldlm-144m/resolve/main/generate.py

python generate.py "The meaning of life is"
python generate.py --steps 20 --temperature 0.5 "Once upon a time"

<details> <summary><b>Generation parameters</b></summary>

Tips:

• Use temperature >= 0.5 — greedy decoding causes mode collapse in diffusion LMs (known artifact).
• More steps improves coherence, especially for longer generations. Try 15-20 for best quality.
• The model generates in blocks of 32 tokens, so generation is naturally "bursty."

</details> <details> <summary><b>Load a specific checkpoint</b></summary>

102 checkpoints are available at 250-step intervals from step 250 to step 25,500:

model = SmolDLM.from_pretrained(
    "HoangHa/smoldlm-144m",
    checkpoint="checkpoints/model_step_025000.safetensors"
)

</details>

How Diffusion LMs Work

Traditional language models (GPT, LLaMA) generate text one token at a time, left to right. Diffusion language models take a fundamentally different approach:

Step 0:  [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK]
Step 1:  [MASK] [MASK]  The   [MASK] [MASK] [MASK]  was   [MASK]
Step 2:  [MASK]  In     The   [MASK] year   [MASK]  was   great
Step 3:   And    In     The   first  year   2024   was   great

1. Start with noise — Fill a block of 32 positions with <|mask|> tokens
2. Predict all positions — The model predicts what token belongs at every masked position simultaneously
3. Unmask the most confident — Reveal positions where the model is most certain
4. Iterate — Re-predict remaining positions with newly revealed context
5. Next block — Cache the completed block and move to the next 32 positions

This parallel-within-block generation enables:

• Self-correction — The model can revise earlier predictions within a block as context builds
• Parallel decoding — Multiple tokens generated per forward pass
• Non-autoregressive flexibility — No strict left-to-right constraint within blocks

Training

Training Curves

Loss, gradient norm, and throughput tracked via Trackio.

<details> <summary><b>Training features</b></summary>

• FlexAttention — Compiled block-sparse staircase masks (PyTorch 2.5+)
• Liger Kernels — Fused RMSNorm + SwiGLU
• FP8 Training — Optional tensorwise dynamic scaling on H100+ (--fp8)
• torch.compile — Per-block compilation with Inductor
• Document Packing — No padding waste, attention masked at document boundaries
• Selective Activation Checkpointing — Save matmul/attention outputs, recompute cheap ops

</details>

Tokenizer

SmolDLM uses a hybrid BPE tokenizer:

• Merges from SmolLM2 cosmo2-tokenizer (49,138 tokens)
• Pre-tokenizer from Qwen3 (NFC normalization + GPT-4 regex + ByteLevel)
• 14 special tokens at IDs 0-13:

Limitations

• Early checkpoint — Trained on ~26B of 100B available tokens (26%). The model has learned language structure but generation quality is limited. Coherent multi-sentence output requires further training.
• Repetition — Like other early-stage diffusion LMs, the model can produce repetitive patterns, especially at low temperatures.
• Greedy decoding degenerates — Temperature=0 causes mode collapse (known dLLM artifact). Always use temperature >= 0.5.
• English only — Trained on English web text.
• Not instruction-tuned — Base model, no chat or instruction-following capability.
• Custom architecture — Not compatible with HuggingFace transformers AutoModel. Use the provided modeling_smoldlm.py.
• No safety training — May generate harmful, biased, or factually incorrect content.

Related Work

SmolDLM builds on ideas from:

• LLaDA (GSAI-ML) — Large-scale diffusion LM proving the paradigm works at 8B
• BD3-LMs (Arriaga et al.) — Block diffusion with staircase attention
• MDLM (Sahoo et al.) — Masked diffusion framework
• SmolLM2 (HuggingFace) — Architecture and tokenizer foundation
• Fast-dLLM (NVIDIA) — Inference acceleration techniques
• Gated Query Attention — Attention sink elimination

License

Apache 2.0

Citation

@misc{smoldlm2026,
  title   = {SmolDLM: A Small Block Diffusion Language Model},
  author  = {Hoang Ha},
  year    = {2026},
  url     = {https://huggingface.co/HoangHa/smoldlm-144m}
}