bert-dllm-wikitext2

RoBERTa dLLM · WikiText-2

This is a research checkpoint that turns roberta-base, a BERT-style bidirectional masked-language model, into a masked discrete language diffusion model (dLLM). It is the checkpoint served by Caffin/bert-dllm.

Training

• Base model: roberta-base
• Dataset: Salesforce/wikitext, wikitext-2-raw-v1, training split
• Canvas length: 256 tokens
• Fixed conditioning prefix: first 16 tokens are never masked
• Noise schedule: one probability sampled per batch from 1.0, 0.9, ..., 0.1
• Loss: original token IDs only at positions selected for masking
• Run: 1,000 steps, batch size 4, learning rate 5e-5, seed 42
• Hardware: local Apple Silicon MPS

Sequences with no token after the fixed prefix are excluded because they contain no position that can contribute to the diffusion loss.

Inference

Start from a 256-token canvas containing the preserved prefix and 240 mask tokens. For ten reverse passes, predict every masked position in parallel, sample replacement tokens, and re-mask 90% down to 0% of the generated canvas. The Space exposes both:

1. Article reproduction: uniform random re-masking.
2. Confidence-aware: retain the most confident positions before the following pass.

The latter is included as a decoding comparison; it is not presented as the exact reproduction mode.

Limitations

This is a short WikiText-2 research fine-tune designed to demonstrate the masked diffusion objective and reverse process. It is not instruction-tuned, may repeat text or produce weak continuations, and should not be treated as a production language model.