Downloads · 30 days
15
2% of all-time downloads
pinkelephant-llm-48b-moe-base
pinkelephantlimited/pinkelephant-llm-48b-moe-base
pinkelephant-llm-48b-moe-base is a text generation model from pinkelephantlimited. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
The sparse foundation behind the Pink Elephant 48B family.
All-time downloads
771
Public
Parameters
47.7B
95.4 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors95.4 GB · 100%
Parameter types
How the weights are stored.
BF1647.7B · 100%
Try a prompt
Base models
Model card
From the Hugging Face model README
Pink Elephant 48B MoE — Upcycled Base
The sparse foundation behind the Pink Elephant 48B family.
This repository hosts the upcycled Mixture-of-Experts (MoE) base from which the
production flagship pinkelephant-llm-48b was trained. It is intended for research and as
a merging foundation. For inference, use the merged production model instead — it is ready
to run out of the box.
How the MoE base was created
The base was created by upcycling the dense 14B foundation of pink-elephant-llm-14b:
its feedforward weights were mathematically split and replicated into eight expert
networks per layer, initialized to reproduce the parent model's behavior exactly.
Model Card (verified facts)
| Specification | Value |
|---|---|
| Parameters | 47,691,289,920 (~47.69 B, bf16) |
| Architecture | Mixture of Experts — 8 experts, top-2 routing |
| Transformer layers | 40 |
| Hidden size | 5,120 |
| MLP / expert hidden size | 8,960 |
| Attention heads | 40 (10 KV heads, head dim 128) |
| Expert networks | 8 per layer (640 expert tensors verified in the weight file) |
| Active per token | 2 of 8 experts (~30% of params) |
| Context window | 16,384 tokens |
| Vocabulary size | 100,352 |
| Precision | bfloat16 (single shard, ~95.4 GB) |
| Router | Randomly initialized (untrained) |
| License | MIT |
Parameter and expert-structure counts are verified directly from the
model.safetensorsheader (640expert*.weighttensors across 40 layers × 8 experts).
Charts
| <img src="moefigs/capacity.png" width="420"><br>Total capacity 47.69B vs ~14.7B active per token (~30.7%) | <img src="moefigs/expert_structure.png" width="400"><br>Verified expert structure — 40 layers × 8 experts |
Note on the router: the router weights in this base are initialized, not trained. A fully trained router is included with the
step-900,step-1000, andfinalcheckpoints ofpinkelephant-llm-48b-adapters(asgate.safetensors), and is already fused into the mergedpinkelephant-llm-48b.
Usage
Load the base for research or as a merging foundation:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"pinkelephantlimited/pinkelephant-llm-48b-moe-base",
torch_dtype="auto", device_map="auto", trust_remote_code=True,
)
Attach the trained adapters on top:
from peft import PeftModel
model = PeftModel.from_pretrained(
model, "pinkelephantlimited/pinkelephant-llm-48b-adapters",
subfolder="adapters/final",
)
Or load the ready-to-run merged model directly:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"pinkelephantlimited/pinkelephant-llm-48b",
torch_dtype="auto", device_map="auto", trust_remote_code=True,
)
Release Family
| Repository | Purpose |
|---|---|
pinkelephant-llm-48b | Merged production model — ready to run |
pinkelephant-llm-48b-moe-base (this repo) | Upcycled MoE base (bf16) — for research and merging |
pinkelephant-llm-48b-adapters | LoRA adapters + trained router, published every 100 steps |
pink-elephant-llm-14b | The dense 14B foundation |
pinkelephant-llm-48b-sft-tokenized | The instruction-tuning dataset (tokenized) |
Documentation
- Release report:
RELEASE_REPORT.pdf
Pink Elephant Limited — publishing large, sovereign language models.
- Task
- Text Generation
- Library
- transformers
- Type
- pinkelephant_moe
- License
- mit
- Languages
- en
- Created
- Aug 9, 2026
- Updated
- Aug 21, 2026