ravaan-diff-70m

Ravaan-DIFF-70M — a masked diffusion language model for Urdu

To our knowledge this is the first masked diffusion language model trained for Urdu (literature searched 2026-09-23; see Novelty below for what that claim does and does not cover). 70M parameters, trained from scratch on a purpose-built 85.4M-token Urdu corpus.

It is half of a matched pair. Its autoregressive twin, ravaan-ar-70m, was trained on the identical corpus with the identical model, task mixture and tokenizer — so the two can be compared without confounds, which is the point of releasing both.

The result

Held-out validation bits-per-byte on native Urdu, lower is better:

The diffusion model wins, and the interesting part is why it was allowed to. Autoregressive training on this corpus turns at 4 epochs — past that, more compute actively makes the model worse, and by 16 epochs it has degraded from 0.7774 to 0.8690. The diffusion model descends monotonically for all 64 epochs and has still not turned: its last doubling bought −0.0386 bpb. The two objectives were given the same corpus and the same money; only one of them could spend it.

This reproduces, for a naturally low-resource non-Latin-script language, the data-constrained crossover reported for English by Prabhudesai et al. (arXiv:2507.15857) and Ni et al. (arXiv:2511.03276).

The full epoch curve

Only the last rung carries an error bar, and that is permanent rather than an oversight: the other six checkpoints were lost when the rented instance was destroyed — the transfer link ran at 19 KB/s against 5.9 GB of checkpoints, so only the final one was saved. Their scores survive because the run scored them before they were lost; the weights do not, so no draw can ever be added to them. The shape of the curve is unaffected. Every one of those six is a single draw of an estimator whose single-draw sd is 0.0054, which is far smaller than the gaps between rungs.

Error bars, and an asymmetry that matters

The diffusion number is an ELBO — an upper bound on the true negative log-likelihood, and it is a Monte Carlo estimate of that bound: the loss draws one masking rate and one mask per sequence, so a single evaluation pass is a sample, not a property of the checkpoint. Every diffusion figure above is the mean of 9 independent seeded draws of the full 4,874-sequence validation split:

urdu bpb 0.7659 ± 0.0018 (sem; single-draw sd 0.0054, range 0.7575–0.7737)

The AR model needs none of this — its likelihood is exact.

That asymmetry runs in this result's favour and the reader should know it. A bound that is lower than an exact likelihood proves the bound's model is better. The reverse would prove nothing. So this comparison can establish a diffusion win and could never have established an AR one — which is a limitation of the instrument, and here it happens to point the right way.

Usage

The architecture is not a transformers model and AutoModel will not load it. Inference code ships in this repository, because for a diffusion model the denoising schedule, the step count and the forbidden tokens are part of the model.

pip install torch sentencepiece safetensors
python generate.py --prefix "پاکستان کی معیشت" --tokens 160

from ravaan_infer.loader import load
from ravaan_infer.sampling import SamplingConfig, build_generator, generate, prompts
import sentencepiece as spm

arm = load(".", device="cpu")
sp = spm.SentencePieceProcessor(model_file=str(arm.tokenizer_path))

prefix = sp.encode("پاکستان کی معیشت")
prompt = prompts.lm(arm.framing, "diff", prefix=prefix,
                    length=len(prefix) + 1 + 160, mask_id=arm.mask_id)
out = generate(
    arm.model, prompt,
    config=SamplingConfig(temperature=1.0, top_p=0.95),
    steps=8, schedule="gumbel", gumbel=2.0,
    forbid=arm.forbidden + (arm.eos_id,),          # see below
    generator=build_generator("cpu", 0),
)
print(sp.decode([int(t) for t in out.tokens[0] if t >= 16]))

The decoder settings are measured, not preferences

Change them and you are using a different model. All three were established by sweep:

• steps=8. More denoising steps decode worse here, monotonically. 8 steps score repetition 0.016; 160 steps score 0.084; the confidence schedule at any step count scores 0.29–0.61 and degenerates into looping one phrase. Measured twice at two scales. 8 steps is also 20× cheaper, so the better setting is the faster one.
• schedule="gumbel", gumbel=2.0. The two schedules originally specified — random and confidence — turn out to be the two limits of one family, and both fail, in opposite directions. The interior is where readable Urdu comes out.
• </s> forbidden. On a fixed-width canvas with no context, end-of-sequence wins 47% of first commits, and the model fills the rest with padding. Forbidding it moves the Arabic-script share of output from 0.000 to 1.000. This is stated here rather than buried in a default.

Does it memorize?

64 epochs is heavy repetition, and an autoregressive model trained that way on this project's other corpus was measured reciting 24.9% of its 32-token windows verbatim. So it was checked, against the training stream, with real held-out Urdu as a control:

Training data

A frozen, decontaminated Urdu corpus of 85,362,688 unique tokens built for this project. The pipeline is ten stages — quality filtering, exact and near-duplicate removal, PII screening, held-out decontamination, and a script-aware split — and every stage carries its own decision record in the source repository.

Mixture, by tokens: 74.3% native Urdu, 20.4% Roman Urdu, 5.4% code-switched.

The tokenizer is a 16,384-piece SentencePiece model trained on this corpus (fingerprint 2855877c8ecd38c9). It ships with the weights, and the model cannot be used with any other one.

Licensing

The released weights are Apache-2.0. Urdu Wikipedia is CC-BY-SA-3.0, and whether share-alike reaches model weights is genuinely unsettled; this release takes the conventional position that weights are not a derivative work of training text, and discloses the input licence rather than relying on the reader not to ask. No raw training text is redistributed — the corpus ships as code, a manifest and checksums, so it can be rebuilt but not downloaded.

Limitations — read this part

These are measurements from this project, not boilerplate.

The Urdu was never read by a native speaker. Every judgement of fluency, grammaticality and sense in this project — including the samples above — was made by an LLM. A human evaluation was designed and specified and never ran: the three annotators it needed were never recruited. Treat every qualitative claim as unvalidated.

A known lexical defect is in the training data, disclosed rather than filtered. The Roman-Urdu source renders eight common Urdu words as fixed unrelated words — کرتے→baghaawat, بس→dehli, گھر→mamu — at 52–93% of their occurrences. That is 2.77% of rows after deduplication, inside the 20.4% of the corpus that is Roman Urdu. It was left in and stated, because filtering the eight confirmed cases cannot reach the unscreened tail without the native-speaker pass that never happened. Transliteration output is the worst affected.

Part of the held-out set is not decontaminated. The Roman-Urdu held-out split carries within-source deduplication only and never went through the containment pass; the native-Urdu and code-switched splits did. Separately, roughly 4.4% of the native-Urdu held-out split is estimated to also occur inside the FineWeb2 training pool. Both inflate held-out scores, and both do so for the two arms equally — so the AR-vs-diffusion comparison is largely robust to them, while the absolute bits-per-byte figures are optimistic.

One seed. Every number here comes from a single training run per arm. There is no estimate of seed-to-seed variance on this pair, so the gap between the two models cannot be separated from one initialization draw. (The diffusion objective was replicated across two seeds on a different, smaller corpus, where the two agreed to 0.29σ — that bounds the objective's stability, not this comparison.)

70M parameters. This is a small research model. It has no instruction tuning, no alignment, no safety filtering, and it will produce fluent-sounding false statements. It is trained on filtered web text and reproduces the biases of that text. It is not a product.

A pretraining coherence gate was recorded as failed. A 25M-parameter pilot was required to produce coherent Urdu before the full runs were funded, and it did not. The gate's kill criterion was "implementation bug — debug, do not scale"; no bug was found, and the gate's premise was judged wrong rather than the code — a 25M model on 7.4M tokens was judged below the scale at which coherent Urdu is reachable at all. That judgement was accepted with the risk named, and it is recorded here as unmet rather than quietly passed.

Novelty

What the literature review supports, searched 2026-09-23:

To our knowledge, no masked diffusion language model has previously been trained for Urdu, and the data-constrained diffusion-vs-autoregressive crossover has not previously been replicated on a naturally low-resource, non-Latin-script natural language.

What it does not support, stated so nobody has to find it out by checking:

• Not the first diffusion LM for a non-English language — see Diffutron (arXiv:2603.20466), a masked diffusion LM for Turkish.
• Not the discovery of the crossover phenomenon, which is Prabhudesai et al. (arXiv:2507.15857) and Ni et al. (arXiv:2511.03276). This is a replication on a language and script they did not test.
• "First" is a claim about a literature search on a date, not a fact about the world. Urdu-language venues (CLE Lahore, LREC regional tracks) are under-indexed by the tools used, and no paywalled venue was searched.

The search covered arXiv, ACL Anthology, OpenReview and the Hugging Face Hub. The only Urdu model on the Hub carrying a diffusion tag is a text-to-speech model, not a language model.

Provenance

Trained on one rented RTX 5090 — 64 epochs, 5.46B tokens processed, 8.2 hours, about $4.60. The whole project, six training runs included, cost roughly $28.

The design was preregistered before any training: four falsifiable predictions, a primary endpoint, and a deviation log that records every departure with its date and reason. Both are in the source repository, along with a findings register of 71 numbered findings — including the bugs, the two abandoned runs, and the measurement that overturned an earlier conclusion in this project's own write-up.

• Source, preregistration, corpus pipeline: https://github.com/AwaisAdilKhokhar/ravaan
• Matched autoregressive model: https://huggingface.co/AwaisAdilKhokhar/ravaan-ar-70m