Downloads · 30 days
16
8% of all-time downloads
puigde/modern-transformer-mha-370M-15B-slimpajama
modern-transformer-mha-370M-15B-slimpajama is a text generation model from puigde. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is for research purposes only and is not intended for production use.
Downloads · 30 days
16
8% of all-time downloads
All-time downloads
198
Public
Parameters
371M
1.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.5 GB · 100%
From the Hugging Face model README
This model is for research purposes only and is not intended for production use.
A modern transformer language model (371.2M parameters) with multi-head attention and modernized architectural features, pretrained from scratch on 15B tokens from SlimPajama.
This model uses the same modernized design choices as the GQA variant -- partial rotary position embeddings, output gating, QK-normalization, and zero-centered RMSNorm -- but with standard multi-head attention instead of grouped-query attention. It serves as a modernized attention baseline for comparison with linear attention and state-space models in the Sequence Modeling Baselines collection.
Trained with flash-linear-attention and Flame.
Requires: pip install flash-linear-attention
import torch
import fla.models # registers the Transformer architecture with HuggingFace Auto classes
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"puigde/modern-transformer-mha-370M-15B-slimpajama",
trust_remote_code=True,
torch_dtype=torch.bfloat16,
).cuda()
tokenizer = AutoTokenizer.from_pretrained(
"puigde/modern-transformer-mha-370M-15B-slimpajama"
)
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=50, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
The import fla.models line registers the Transformer architecture with HuggingFace's Auto classes. Without it, from_pretrained will fail with an unknown model type error.
| Parameters | 371,247,616 |
| Layers | 22 |
| Hidden size | 1,024 |
| Heads | 8 (MHA) |
| KV heads | 8 |
| Head dim | 128 |
| Rotary dim | 64 (50% partial RoPE) |
| Output gate | Sigmoid |
| QK norm | Yes |
| Normalization | Zero-centered RMSNorm |
| FFN | SwiGLU, 4x hidden ratio |
| Vocab size | 32,000 |
| Context length | 2,048 |
| Tied embeddings | No |
| Dataset | cerebras/SlimPajama-627B, train split |
| Tokens | 15,032,385,536 |
| Steps | 28,610 |
| Batch size | 256 sequences (8/GPU x 8 GPUs x 4 grad accum) |
| Sequence length | 2,048 |
| Optimizer | AdamW (fused), betas=(0.9, 0.95), eps=1e-15 |
| Learning rate | 4e-4 peak, cosine to 4e-5 |
| Warmup | 1,024 steps |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Precision | bfloat16 compute, float32 reduce |
| Hardware | 8x NVIDIA A100-SXM4-40GB |
| Training time | ~15.3 hours |
| Final loss | 2.508 |
| Seed | 42 |
Tokenizer: LlamaTokenizer (from fla-hub/gla-1.3B-100B), vocab 32,000.
Zero-shot, lm-evaluation-harness:
| Benchmark | Metric | Score |
|---|---|---|
| HellaSwag | acc_norm | 36.1 |
| PIQA | acc_norm | 66.2 |
| ARC-Easy | acc | 46.9 |
| ARC-Challenge | acc_norm | 22.3 |
| WinoGrande | acc | 52.7 |
| LAMBADA | acc | 34.0 |
| BoolQ | acc | 58.9 |
| COPA | acc | 66.0 |
| SciQ | acc | 80.3 |
| OpenBookQA | acc_norm | 30.2 |
| WikiText-2 | word_ppl | 25.9 |
RULER (needle-in-a-haystack):
| Task | 1K | 2K | 4K |
|---|---|---|---|
| Single-1 | 1.00 | 1.00 | 0.68 |
| Single-2 | 1.00 | 1.00 | 0.62 |
| Single-3 | 1.00 | 1.00 | 0.18 |
| Multi-key-1 | 0.89 | 0.86 | 0.22 |
Note: Unlike the GQA variant which collapses at 4K, MHA retains partial performance beyond the 2K training context, likely due to higher per-head attention capacity with 8 heads at 128 dims vs 4 heads at 256 dims.
Recall:
| Task | Score |
|---|---|
| SWDE | 0.62 |
| FDA | 0.24 |
@article{yang2024fla,
title={Gated Linear Attention Transformers with Hardware-Efficient Training},
author={Yang, Songlin and Wang, Bailin and Shen, Yikang and Panda, Rameswar and Kim, Yoon},
journal={arXiv preprint arXiv:2312.06635},
year={2024}
}