Downloads · 30 days
5
20% of all-time downloads
mowe-moe-768-19l
FAIRC/mowe-moe-768-19l
mowe-moe-768-19l is a machine learning model from FAIRC. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Inference checkpoint for the standard mixture-of-experts baseline released with Mixture of Wide Experts. Every selected expert reads and writes the full model width.
All-time downloads
25
Public
Parameters
364M
1.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors1.5 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 6, 2026
- Updated
- Aug 6, 2026
- SHA
- b3884c3e
Model card
From the Hugging Face model README
Standard MoE 768, 19 Layers
Inference checkpoint for the standard mixture-of-experts baseline released with Mixture of Wide Experts. Every selected expert reads and writes the full model width.
| Property | Value |
|---|---|
| Parameters | 363.71M |
| Transformer layers | 19 |
| Model width | 768 |
| Attention heads | 12 |
| Experts | 768 |
| Hidden units per expert | 8 |
| Selected experts per token | 38 |
| Context length | 1,024 |
| Tokenizer | GPT-2 |
The checkpoint is an inference-only safetensors state dictionary. It contains
no optimizer state. This is a base language model and is not instruction-tuned.
Loading
Install the companion MoWE extension for OpenLanguageModel, then load this repository directory:
import torch
from mowe import load_checkpoint
model = load_checkpoint(
"/path/to/moe-768-19l",
device="cuda",
dtype=torch.bfloat16,
).eval()
Limitations
This is a small research model. It may produce incorrect, biased, unsafe, or memorized text and should not be used as a production assistant without separate evaluation and safeguards.
License
Released under the Apache License 2.0. Please cite the accompanying MoWE paper.
- License
- apache-2.0
- Created
- Aug 6, 2026
- Updated
- Aug 6, 2026