Llama3.2-Mamba2-3B-distill

JunxiongWang/Llama3.2-Mamba2-3B-distill

Llama3.2-Mamba2-3B-distill is a machine learning model from JunxiongWang. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

Zero-shot results when using the Llama-3.1-70B-Instruct as the teacher model, and the Llama-3.2-3B-Instruct as the initialized model

Downloads · 30 days

1% of all-time downloads

All-time downloads

7.2K

Public

Repo size

26.8 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin13.4 GB · 100%

At a glance

License: apache-2.0
Model type: llama
Access: Public
Created: Oct 15, 2024
Updated: Nov 17, 2024
SHA: 2373e132

Model card

From the Hugging Face model README

Zero-shot results when using the Llama-3.1-70B-Instruct as the teacher model, and the Llama-3.2-3B-Instruct as the initialized model

Model	Llama-3.2-3B-Instruct	Llama3.2-Mamba-3B-distill	Llama3.2-Mamba-3B-dpo	Llama3.2-Mamba2-3B-distill	Llama3.2-Mamba2-3B-dpo
Initialization Model	N/A	Llama-3.2-3B-Instruct	Llama-3.2-3B-Instruct	Llama-3.2-3B-Instruct	Llama-3.2-3B-Instruct
Teacher Model	N/A	Llama-3.1-70B-Instruct	Llama-3.1-70B-Instruct	Llama-3.1-70B-Instruct	Llama-3.1-70B-Instruct
arc_challenge	0.459	0.4838	0.5265	0.4667	0.541
arc_easy	0.7407	0.7765	0.7997	0.7668	0.8026
hellaswag	0.7043	0.7037	0.7256	0.6913	0.7445
mmlu	0.6043	0.5448	0.5509	0.5312	0.5247
openbookqa	0.36	0.394	0.416	0.388	0.424
piqa	0.7568	0.7731	0.7731	0.7601	0.7769
pubmedqa	0.696	0.664	0.7	0.638	0.654
race	0.4067	0.4029	0.4364	0.3981	0.4344
winogrande	0.6748	0.6732	0.674	0.6606	0.6732
truthfulqa	0.3801	0.4202	0.4853	0.3478	0.5028

@article{junxiongdaniele2024mambainllama,
  title   = {The Mamba in the Llama: Distilling and Accelerating Hybrid Models},
  author  = {Junxiong Wang and Daniele Paliotta and Avner May and Alexander M. Rush and Tri Dao},
  journal = {arXiv preprint arXiv:2408.15237},
  year    = {2024}
}

Type: llama
License: apache-2.0
Created: Oct 15, 2024
Updated: Nov 17, 2024