Downloads · 30 days
97
4% of all-time downloads
MarxistLeninist/AGILLM-GB10-1PF
AGILLM-GB10-1PF is a machine learning model from MarxistLeninist. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as apache-2.0.
- Parameters: 1,990,038,784 conventional parameters (~1.990B) - Training target: 600 billion tokens - Objective: autoregressive next-token prediction only - Position scheme: NoPE, with causal/window structure only - P…
Downloads · 30 days
97
4% of all-time downloads
All-time downloads
2.6K
Public
Repo size
606 GB
Likes
0
Public
Click a slice to open those files.
.ckpt566 GB · 93%
From the Hugging Face model README
recovery/rpv16-affine-step00001856.ckptA ~2B-parameter language-model architecture designed around NVIDIA GB10 / SM121 sparse NVFP4 from the start, rather than retrofitting a generic Transformer.
mxf4nvf4 SM121 sparse MMA is the target hot pathThe architecture is machine-readably frozen in ARCHITECTURE_LOCK.json.
GB10 has an unusually large 128 GB GPU-addressable unified-memory pool but far less memory bandwidth than its sparse FP4 tensor-core issue capability would suggest. The model therefore tries to turn language-model training into large, repeatedly reusable sparse NVFP4 GEMMs instead of making the GPU shuffle small matrices and optimizer state all day.
The key expert shape, 1280 x 5120, comes from direct measurements on this GB10 and matches the existing paired-4:8 CUTLASS sparse runtime. It is not a copied Llama ratio.
The repository reports these separately:
No synthetic filler FLOPs are counted as useful model work.
Verified historical starting evidence on this GB10 was 969.794609 TFLOP/s best. After stopping all unrelated GPU compute and autotuning launch geometry/register pressure, the current checked peak is 992.017989 TFLOP/s, or 99.20% of 1 PFLOP/s, at 34 accumulators, 20 blocks/SM, 128 threads and 1200 iterations. The Vast container is denied permission to raise the GPU locked clock, so the remaining gap is presently host-clock limited. Search receipts are in evidence/1pf_search/.
NVIDIA's ~1 PFLOP figure is a sparse FP4 hardware ceiling, not a claim that an end-to-end training step can execute at 1 PFLOP/s. A 2 PFLOP number, if ever reported here, must be explicitly labelled algorithmic/effective useful compute and never confused with actual single-GB10 tensor issue throughput.
The model has already completed real training steps on HuggingFaceFW/fineweb-edu using the bundled tokenizer, causal cross-entropy, sparse forward kernels, backward propagation and AdamW.
Initial full-gradient canary:
A two-microbatch accumulation test reused packed weights: the second microbatch added 288 sparse kernel calls but only 9 new packs, demonstrating the intended packed-weight-residency behavior. The subsequent stage-local schedule reduced live CUDA allocation substantially while retaining a real final-CE gradient for the selected stage.
Receipts are under evidence/canary/.
agillm_gb10_1pf.py contains a streaming Hugging Face dataset loader. The intended 600B run uses a weighted mixture including FineWeb-Edu, FineWeb, Wikipedia, C4, Cosmopedia, Proof-Pile-2, Dolma and code sources. Source failures are allowed to fail over rather than stall training.
python3 agillm_gb10_1pf.py profile
python3 agillm_gb10_1pf.py dataset-probe --dataset fineweb-edu
python3 agillm_gb10_1pf.py train --dataset fineweb-edu --batch 1 --seq 512 --steps 2 --grad-accum 2
The repository contains the exact tokenizer bundle and the current GB10 sparse runtime binaries plus CUDA source used to rebuild/audit them.
Source, target contracts, tokenizer, kernel sources, reproducibility receipts and promoted checkpoints belong here. Transient profiler dumps and every throwaway checkpoint do not. Hugging Face is a model repository, not a landfill with Git LFS.
The upstream peak-MMA reproducer under third_party/gb10_peak_mma/ retains its BSD-3-Clause license and commit metadata. AGILLM-GB10-1PF source outside third-party material is Apache-2.0 unless a file says otherwise.
The active GB10 training architecture now uses NoPE, exact 8,192 routed rows per expert for the 24×2048 production geometry, one bounded native sparse context per layer, and checkpointed exact tied-vocabulary CE to avoid materialising the full 49,152×129,280 logit matrix. This reduced CUDA reserved memory from roughly 95.8 GB to roughly 24–36 GB during early stage rotation while improving warm microstep throughput into the ~19–23k tok/s range. Sparse packs remain stable between stage changes rather than growing every microstep.
Training uses a 17-slot hardware-native update cycle: stages 15→0 one at a time, followed by one exact tied token-interface anchor. No update makes all ~1.987B parameters trainable simultaneously. The target remains 600B training tokens.
The flat 129,280-way tied-softmax head was replaced by a rank-16 affine product vocabulary. Since 129,280 = 505 × 256, each mixture component predicts a normalized 505-way factor and 256-way factor; each rank uses a different bijective affine permutation of token IDs before factorization. The mixture therefore remains a proper normalized distribution over all 129,280 token IDs while reducing the output-head compute width by roughly 10.5× versus the flat classifier.
The decoder was migrated with seven head-only optimizer updates from step 896 to step 952 before local stage training resumed. Loss moved from roughly 11.787 to 11.683 during that migration. A complete post-warm 17-phase cycle sustained 42,018.5 training tok/s end-to-end. Batch-size tests placed the full-graph throughput knee at batch 24 / 8,192 routed rows per expert: batch 12 (M=4096) and batch 48 (M=16384) were both slightly slower per token.
Benchmark and migration receipts are under evidence/rpv16/.
Production now quantizes each expert input once for gate + up instead of twice. Gate and up still use separate packed sparse weights, independent FP4 weight scales, and their ordinary native sparse GEMMs; only the identical activation quantization is shared. An 8,192-row expert A/B was bitwise identical for output, dX, and all gate/up/down weight gradients. Frozen-expert forward improved from 5.634 ms to 5.216 ms (+8.0%). A matched full stage-4 replay improved median throughput from 54,956 to 56,218 tok/s (+2.30%).
After live promotion from checkpoint 2048, the same stage-14 eight-microstep window improved from 30,188.7 to 31,515.2 tok/s (+4.39%), with exact 8,192-row routing and a successful optimizer update. The first shared-quant recovery checkpoint was written at step 2112 / 103,809,024 tokens. The previously published 42,018.5 tok/s full-cycle figure predates shared-quant; a new complete-cycle figure is not claimed until measured. Receipts are under evidence/sharedquant/.