Downloads · 30 days
0
experimental_gqa_1_5b
Norapom/experimental_gqa_1_5b
experimental_gqa_1_5b is a machine learning model from Norapom. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for megatron-lm. The card lists the license as other.
Megatron-LM checkpoint, trained from scratch on FineWeb sample-10BT/100BTpart1 (text) + codeparrot-clean (code), tokenized with cl100kbase.
Access
Public
Updated May 17, 2026
Repo size
94.5 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.distcp23.6 GB · 100%
At a glance
- Library
- megatron-lm
- License
- other
- Access
- Public
- Created
- May 8, 2026
- Updated
- May 17, 2026
- SHA
- 9892cdef
Model card
From the Hugging Face model README
experimental_gqa_1_5b — iter 16,000
Megatron-LM checkpoint, trained from scratch on FineWeb sample-10BT/100BT_part1 (text) + codeparrot-clean (code), tokenized with cl100k_base.
This branch: iter 16,000 (~14.68B tokens trained). Other revisions: branches , , , . The branch tracks iter 16,000.
Architecture
- Layers: 32, hidden 2048, FFN 4096, GQA 16Q/4KV, head_dim 128
- Vocab 100,352 (cl100k_base, padded for TP=4)
- RoPE base 1e7, partial 0.25; SwiGLU; RMSNorm + 1p
- Attention output gate; QK-LayerNorm with WD; untied embeddings
- Pretrain: bf16
Loading
This is a Megatron-LM checkpoint (sharded). To use: Updated Git hooks. Git LFS initialized. Then point Megatron's at the cloned dir; tokenizer is at . Architecture flags must match — see the recipe in the upstream training repo.
Training schedule
- Iters 0–12,000: GBS=128, LR 3e-4 cosine, warmup 1000 — over 80,000-step schedule
- Iters 12,000–16,000: GBS=512, LR cosine warm-restart over 30,000-step schedule
Tokenizer (chat-extended)
See tokenizer/ for the Cl100kChatTokenizer with 16 reserved chat/think/tool tokens (IDs 100277-100292).
- Library
- megatron-lm
- License
- other
- Languages
- en
- Created
- May 8, 2026
- Updated
- May 17, 2026