Downloads · 30 days
28
13% of all-time downloads
Foundry-LLM-1.2B-1T
TRI-ML/Foundry-LLM-1.2B-1T
Foundry-LLM-1.2B-1T is a text generation model from TRI-ML. Use it when you need the model to write or continue text. It is set up for vla-foundry. The card lists the license as apache-2.0.
A 1.2B parameter language model pretrained on 1T tokens, part of the VLA Foundry collection.
All-time downloads
210
Public
Repo size
5.7 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.pt5.7 GB · 100%
At a glance
- Task
- Text Generation
- Library
- vla-foundry
- License
- apache-2.0
- Access
- Public
- Created
- Apr 16, 2026
- Updated
- Apr 23, 2026
- SHA
- 1e71bc75
Try a prompt
Model card
From the Hugging Face model README
Foundry-LLM-1.2B-1T
A 1.2B parameter language model pretrained on 1T tokens, part of the VLA Foundry collection.
Model Description
- Architecture: Transformer (24 layers, 2048 hidden dim, 16 heads, SwiGLU FFN, RoPE, QK-norm)
- Parameters: 1.2B (non-embedding)
- Tokenizer: SmolVLM2 (vocab size 49,280)
- Training data: 1T tokens from DCLM-Baseline-1.0
- LR schedule: Warmup + constant for 800B tokens, then 200B tokens of cosine decay
- Sequence length: 2048
Continuation of Foundry-LLM-1.2B-800B with an additional 200B tokens of cosine-decayed training.
Evaluation Results
Multiple-choice reasoning benchmarks:
| HellaSwag | MMLU | ARC-e | ARC-c | PIQA | WinoGrande | OpenBookQA | BoolQ |
|---|---|---|---|---|---|---|---|
| 66.7 | 26.6 | 71.7 | 39.3 | 77.5 | 62.6 | 40.8 | 65.4 |
Usage
git clone https://github.com/TRI-ML/vla_foundry.git
cd vla_foundry
pip install -e .
from vla_foundry.models.base_model import BaseModel
model = BaseModel.from_pretrained("TRI-ML/Foundry-LLM-1.2B-1T")
Links
- Project page: tri-ml.github.io/vla_foundry
- Paper: VLA Foundry (arXiv 2604.19728)
- Code: github.com/TRI-ML/vla_foundry
- Collection: VLA Foundry collection
- Task
- Text Generation
- Library
- vla-foundry
- License
- apache-2.0
- Created
- Apr 16, 2026
- Updated
- Apr 23, 2026