Downloads · 30 days
0
DZJnsn/attention-is-all-you-need-reproduction
attention-is-all-you-need-reproduction is a machine learning model from DZJnsn. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A reproduction of the original Transformer described in Vaswani et al. (2017), "Attention Is All You Need."
Downloads · 30 days
0
Access
Public
Updated Sep 20, 2026
Repo size
444 MB
Likes
1
Public
Click a slice to open those files.
.pt444 MB · 100%
From the Hugging Face model README
A reproduction of the original Transformer described in Vaswani et al. (2017), "Attention Is All You Need."
Can the results of the original 2017 Transformer paper be reproduced using a modern implementation while remaining as faithful as possible to the original architecture and training procedure?
The reproduction included experiments examining:
| Run | Change | BLEU |
|---|---|---|
| #2 | Correct embedding initialization | 24.30 |
| #3 | + Gradient clipping | 24.65 |
| #4 | + Global loss normalization | 24.37 |
| #5 | Period-correct News Commentary v9 | 25.29 |
| #6 | fp32 training | 25.52 |
| #7 | Paper-like checkpoint averaging window | 25.54 |
| Original paper | Transformer base | 27.3 |
The best completed reproduction reached 25.54 BLEU, compared with 27.3 BLEU reported by the original paper, leaving a gap of 1.76 BLEU.
Run #7 changed the checkpoint-averaging window to more closely match the procedure described in the original paper. The last five checkpoints spanned 6,000 training steps (6% of training), compared with approximately 24% in the earlier runs. This improved BLEU from 25.29 to 25.54.
The trained checkpoint from Run #7 is available in this repository:
This is the five-checkpoint average that produced the best measured result in the reproduction experiments.
The 25.54 BLEU result uses the repository's default 3-gram repetition blocker during decoding. The original 2017 paper did not use this modern decoding aid. In Run #5, disabling the blocker reduced BLEU from 25.29 to 25.15. An equivalent no-blocker evaluation of the Run #7 checkpoint has not yet been measured.
The checkpoint contains the model state dictionary and records the five source checkpoints used to construct the average. Because tied weights can appear under multiple keys in a PyTorch state dictionary, summing all serialized tensor elements does not represent the model's unique parameter count.
Run #7 also provided a reproducibility check on the training pipeline.
Using the same seed, data, and training configuration as Run #5 produced an identical training trajectory across 100,000 steps, even using bf16 and distributed training with DDP.
This means the measured differences between these controlled runs can be attributed to the experimental changes rather than run-to-run nondeterminism for this seed.
Full write-up:
Reproducing "Attention Is All You Need"
https://github.com/fastestapp/DuplicateAttention
The goal of this project was not simply to implement a modern Transformer, but to reproduce the original 2017 experiment as closely as practical.
Where details in the paper were ambiguous or where modern hardware/software required implementation choices, those decisions were treated as experiments rather than silently changing the training procedure.
The reproduction therefore evolved through a series of controlled runs examining gradient clipping, loss normalization, training-data preparation, checkpoint averaging, and evaluation methodology.
The full reasoning behind these experiments, including approaches that did not improve the results, is documented in the accompanying article.
Several seemingly small details mattered.
A recurring pattern was that reproducing what was actually done in 2017 mattered more than adding modern improvements.
"Attention Is All You Need" is one of the foundational papers of modern generative AI. Reading the paper explains the architecture. Reproducing it exposes a different set of questions: which details matter in practice, which assumptions are underspecified, and how closely the reported results can actually be reproduced.
There's a big difference between understanding how a Transformer works and actually building and training one from the original paper.