Downloads · 30 days
7
27% of all-time downloads
mowe-mowe-768-19l
FAIRC/mowe-mowe-768-19l
mowe-mowe-768-19l is a machine learning model from FAIRC. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Inference checkpoint for the Mixture of Wide Experts model. Every selected expert reads the complete residual representation and writes only to its fixed private output coordinate.
All-time downloads
26
Public
Parameters
274M
1.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors1.1 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 6, 2026
- Updated
- Aug 6, 2026
- SHA
- 6e7c29a1
Model card
From the Hugging Face model README
MoWE 768, 19 Layers
Inference checkpoint for the Mixture of Wide Experts model. Every selected expert reads the complete residual representation and writes only to its fixed private output coordinate.
| Property | Value |
|---|---|
| Parameters | 274.17M |
| Transformer layers | 19 |
| Model width | 768 |
| Attention heads | 12 |
| Experts | 768 |
| Hidden units per expert | 8 |
| Selected experts per token | 38 |
| Context length | 1,024 |
| Tokenizer | GPT-2 |
The checkpoint is an inference-only safetensors state dictionary. It contains
no optimizer state. This is a base language model and is not instruction-tuned.
Loading
Install the companion MoWE extension for OpenLanguageModel, then load this repository directory:
import torch
from mowe import load_checkpoint
model = load_checkpoint(
"/path/to/mowe-768-19l",
device="cuda",
dtype=torch.bfloat16,
).eval()
Pass inference_backend="optimized" to use the optional forward-only CUDA and
Triton implementation released with the source repository.
Limitations
This is a small research model. It may produce incorrect, biased, unsafe, or memorized text and should not be used as a production assistant without separate evaluation and safeguards. The optimized kernel path is specialized to this released architecture.
License
Released under the Apache License 2.0. Please cite the accompanying MoWE paper.
- License
- apache-2.0
- Created
- Aug 6, 2026
- Updated
- Aug 6, 2026