pinkelephant-llm-48b

Pink Elephant 48B — The Sparse Intelligence Engine

48 billion parameters of knowledge. A quarter of the compute. One model.

Pink Elephant 48B is a production-ready Mixture-of-Experts (MoE) large language model and the flagship release from Pink Elephant Limited. Upcycled from the proven Pink Elephant 14B dense foundation and instruction-tuned across code and mathematics, it delivers the reasoning depth of a dense 48-billion-parameter network while activating only ~30% of its parameters for every token — 47.7B parameters of total capacity, ~14.7B active at inference.

This repository ships the fully merged, ready-to-run model: the trained LoRA expert adapters and the learned router are fused directly into the base weights. No adapter setup, no PEFT plumbing — just load and generate.

The central promise: every token is routed to a hand-picked council of specialist experts — dense-scale intelligence at a fraction of the compute.

---

The Architecture: A Council of Experts

Pink Elephant 48B replaces the single monolithic feedforward network of a conventional transformer with a cabinet of eight expert networks per layer, orchestrated by a learned router.

• 8 experts per layer — each with its own specialized weights and perspective
• Top-2 sparse routing — the router selects the two most qualified experts for every token; the other six rest
• A trained router — fine-tuned end-to-end alongside the experts, so assignments sharpen continuously with every step

Every token therefore travels through only 2 of 8 experts. This is sparse computing at its most elegant: maximum capability, minimum energy, delivered in real time.

<p align="center"> <img src="figures/architecture.png" alt="Sparse MoE architecture" width="88%"> </p> <p align="center"><em>Figure 1 — The Pink Elephant 48B sparse MoE transformer block: a learned router selects 2 of 8 expert networks for every token.</em></p>

---

48B of Knowledge, a Quarter of the Compute

• Capacity of a dense 48B — a vast reservoir of knowledge and reasoning, spread across specialists that never compromise on depth
• ~3.2× compute savings — with only ~30% of parameters active per token, inference costs a fraction of a dense model of the same scale
• No quality compromise — dense-scale capacity and sparse-scale speed, in a single deployable artifact

<p align="center"> <img src="figures/parameters.png" alt="Parameter efficiency" width="82%"> </p> <p align="center"><em>Figure 2 — Only ~30% of the network activates per token: dense-scale capacity at 3.2× lower compute.</em></p>

---

Trained to Write Code and Reason Mathematically

Pink Elephant 48B was instruction-tuned on ~100,000 curated examples spanning two of the most demanding reasoning domains:

• Software engineering — executable code generation and instruction following (self-oss-instruct, 49,661 examples)
• Mathematical reasoning — rigorous step-by-step problem solving (OpenR1 Math, 50,000 examples)

Training: 1,000 optimization steps, AdamW with a cosine learning-rate schedule (peak 1.5e-4), LoRA (r=32, α=64) applied to the expert projections, with the router trained in full precision.

<p align="center"> <img src="figures/training_loss.png" alt="Training loss and learning-rate schedule" width="90%"> </p> <p align="center"><em>Figure 3 — Training and validation loss over 1,000 optimization steps, with the cosine learning-rate schedule (left-to-right panels).</em></p>

---

An Elegant Scaling Story

Pink Elephant 48B was upcycled, not trained from scratch. The feedforward weights of the established pink-elephant-llm-14b dense foundation were mathematically split and replicated into eight complementary experts — preserving the parent model's knowledge while multiplying capacity roughly threefold. Scaling through intelligence, not brute force.

<p align="center"> <img src="figures/lineage.png" alt="Product lineage" width="90%"> </p> <p align="center"><em>Figure 4 — Product lineage: from the dense 14B foundation to the upcycled MoE base, refined through SFT, and shipped as the merged 48B.</em></p>

---

Getting Started

Load the model directly — no adapters required:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pinkelephantlimited/pinkelephant-llm-48b"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",            # or "cuda" / "cpu"
    trust_remote_code=True,
)

chat = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
prompt = tok.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
with torch.inference_mode():
    out = model.generate(
        prompt.to(model.device),
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )
print(tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True))

Lower-footprint inference

For consumer GPUs, the model can be loaded in 4-bit using standard transformers tooling (requires bitsandbytes):

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    device_map="auto",
    trust_remote_code=True,
)

Note: this model uses a custom architecture shipped in the repository (modeling_pink_elephant.py). Loading it requires trust_remote_code=True.

---

The Refined Successor: 48B-S

The Pink Elephant journey continues. pinkelephant-llm-48b-s takes this flagship through a complete second epoch (steps 1,557 → 3,114), cutting training loss 32% (0.0616 → 0.0419) and merging into a single ready-to-run artifact. If you are starting fresh, use the 48B-S.

<p align="center"> <img src="figures/lineage.png" alt="Product lineage" width="90%"> </p> <p align="center"><em>Product lineage: from the dense 14B core to the upcycled MoE base, refined through SFT, and shipped as the merged 48B — then refined again into the 48B-S.</em></p>

Release Family

---

Evaluation

Benchmark results for the family are published on their respective model cards. The 48B-S refined flagship reports 70.12% pass@1 on HumanEval and 38.40% on MATH-500 (both greedy), with per-problem records under evals/ in that repository. The 14B lineage reports 71.34% pass@1 on HumanEval and 63.40% on MATH-500. The 48B-S HumanEval figure demonstrates the same performance band as the parent at roughly a third of the per-token compute.

---

Verification & Test Report

The full release and verification documentation is published alongside this model:

What was tested

The published artifact was downloaded, loaded, and exercised end-to-end on an NVIDIA RTX PRO 6000 Blackwell Server Edition (102 GB VRAM, 160 GB RAM, PyTorch 2.11.0+cu130, transformers 5.14.1) at native bfloat16 — no quantization, no CPU offload:

Functional smoke test (greedy, 64 tokens)

Every result above is reproducible via the step-by-step recipe in the Verification Test Report (Chapter 11), which requires only a GPU with ≥ 102 GB VRAM and this repository.

---

Limitations

• Fine-tuned primarily on code and mathematical reasoning; general-knowledge coverage inherits from the 14B parent
• Long-horizon, multi-step agentic workflows may benefit from explicit tool-use training
• Outputs should be reviewed before use in regulated or safety-critical decisions

---

License & Commercial Use

Released under the MIT License — free for commercial and research use, modification, and redistribution, with or without attribution.

---

Pink Elephant Limited — publishing large, sovereign language models.