echo-mdlm-341m-ternary-fromscratch

These are research artifacts accompanying the paper Native Ternary Quantization-Aware Training for Masked Diffusion Language Models. They are 341M-parameter masked-diffusion language models trained on Italian FineWeb-2 with a 32k SentencePiece tokenizer. This is not a production model. At roughly 12 tokens per parameter neither the ternary nor the full-precision model composes fluent text; free generation degenerates identically at both precisions. Use these checkpoints for reproduction, infilling analysis, and as paired baselines, not as downstream generators.

• Architecture: bidirectional masked-diffusion transformer, d_model 1024, 24 layers, 16 heads, d_ff 2816, tied embeddings, 32001 vocabulary (mask token id 32000).
• Format: bfloat16, verified to reproduce the full-precision evaluation within 0.0003 masked-CE of the fp32 master.
• Code and reproduction: https://github.com/lupodevelop/echo-1.58
• Tokenizer: spm_it.model (included).

Evaluation (common protocol, sequence length 1024, identical seeded masks)

Echo-1.58 341M, Ternary, From-Scratch Recipe (Negative Control)

A ternary model trained from the first step with the full recovery recipe (distillation plus per-channel scale) for the full 4B-token budget. This is a released negative control. At 341M the recipe applied from step 0 recovers nothing measurable: the model lands at masked-CE 4.9878, within noise of the bare ternary baseline (4.9852), against the 70% recovery the identical recipe delivers at 27M. Post-hoc distillation scales; from-scratch distillation does not. Released because reproducible negative results are rarely shared and are useful for anyone studying scale-dependent distillation.