Downloads · 30 days
4
17% of all-time downloads
mowe-dense-768-19l
FAIRC/mowe-dense-768-19l
mowe-dense-768-19l is a machine learning model from FAIRC. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Inference checkpoint for the dense baseline released with Mixture of Wide Experts.
All-time downloads
23
Public
Parameters
103M
414 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors414 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 6, 2026
- Updated
- Aug 6, 2026
- SHA
- 9fd8c702
Model card
From the Hugging Face model README
Dense 768, 19 Layers
Inference checkpoint for the dense baseline released with Mixture of Wide Experts.
| Property | Value |
|---|---|
| Parameters | 103.46M |
| Transformer layers | 19 |
| Model width | 768 |
| Attention heads | 12 |
| FFN hidden width | 455 |
| Context length | 1,024 |
| Tokenizer | GPT-2 |
The checkpoint is an inference-only safetensors state dictionary. It contains
no optimizer state. This is a base language model and is not instruction-tuned.
Loading
Install the companion MoWE extension for OpenLanguageModel, then load this repository directory:
import torch
from mowe import load_checkpoint
model = load_checkpoint(
"/path/to/dense-768-19l",
device="cuda",
dtype=torch.bfloat16,
).eval()
Limitations
This is a small research model. It may produce incorrect, biased, unsafe, or memorized text and should not be used as a production assistant without separate evaluation and safeguards.
License
Released under the Apache License 2.0. Please cite the accompanying MoWE paper.
- License
- apache-2.0
- Created
- Aug 6, 2026
- Updated
- Aug 6, 2026