Downloads · 30 days
81
0% of all-time downloads
FusionNet_34Bx2_MoE
TomGrc/FusionNet_34Bx2_MoE
FusionNet_34Bx2_MoE is a text generation model from TomGrc. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
Fine-tuned model on English language using MoE method. The FusionNet34Bx2MoE is a model to experiment with the MoE method, which could significantly increase the performance of the original model. The FusionNet34Bx2Mo…
All-time downloads
34.5K
Public
Parameters
60.8B
122 GB on disk
Likes
9
Public
Repo makeup
Click a slice to open those files.
.safetensors122 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Model type
- mixtral
- Access
- Public
- Created
- Jan 21, 2024
- Updated
- Mar 4, 2024
- SHA
- a077e8da
Try a prompt
Model card
From the Hugging Face model README
FusionNet_34Bx2_MoE
Fine-tuned model on English language using MoE method.
Model description
The FusionNet_34Bx2_MoE is a model to experiment with the MoE method, which could significantly increase the performance of the original model. The FusionNet_34Bx2_MoE has 60.8B parameters, and this model is fine-tuned. Enjoy!
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("TomGrc/FusionNet_34Bx2_MoE")
model = AutoModelForCausalLM.from_pretrained("TomGrc/FusionNet_34Bx2_MoE")
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 77.07 |
| AI2 Reasoning Challenge (25-Shot) | 72.95 |
| HellaSwag (10-Shot) | 86.22 |
| MMLU (5-Shot) | 77.05 |
| TruthfulQA (0-shot) | 71.31 |
| Winogrande (5-shot) | 83.98 |
| GSM8k (5-shot) | 70.89 |
- Task
- Text Generation
- Library
- transformers
- Type
- mixtral
- License
- mit
- Languages
- en
- Created
- Jan 21, 2024
- Updated
- Mar 4, 2024