Downloads · 30 days
17
7% of all-time downloads
eagle3-llama-3.1-8b-instruct
jialefu/eagle3-llama-3.1-8b-instruct
eagle3-llama-3.1-8b-instruct is a machine learning model from jialefu. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
This model is a refined version of the original EAGLE-3 model, trained with several key improvements.
All-time downloads
242
Public
Parameters
950M
1.9 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors1.9 GB · 100%
Parameter types
How the weights are stored.
BF16950M · 100%
Base models
Model card
From the Hugging Face model README
Model Description
This model is a refined version of the original EAGLE-3 model, trained with several key improvements.
Features
- Attention Mask Fix: Addresses an issue with the attention mask found in the original EAGLE repository. Further details are available in this pull request.
- Positional Embedding Alignment: The
rope_thetahas been set to 500,000 to align with the Llama-3.1-8B-Instruct model, correcting a mismatch from the original training setting (10,000). - Extended Context Length: The model was trained on data with a sequence length of 4096, an increase from the original 2048. Additionally,
max_position_embeddingsis set to 128,000 to facilitate further pretraining on long contexts. - Training Framework: The model was trained using the SpecForge library.
Benchmarks
Performance was evaluated using the SpecForge benchmark suite.
| Checkpoint | MT-Bench | GSM8K | HumanEval |
|---|---|---|---|
| Original | 5.690 | 6.145 | 6.817 |
| This work | 5.999 | 6.221 | 6.804 |
Training Details
- Epochs: 15
- RoPE Theta: 500,000
- Batch Size: 1
- Learning Rate: 5e-5
- Max Sequence Length: 4096
- Type
- llama
- License
- mit
- Languages
- en
- Created
- Oct 8, 2025
- Updated
- Oct 8, 2025