pinkelephant-llm-48b-adapters

Pink Elephant 48B — LoRA Adapters & Trained Router

The training journey of Pink Elephant 48B, published checkpoint by checkpoint.

This repository hosts the parameter-efficient fine-tuning artifacts for the Pink Elephant 48B MoE model: LoRA adapters trained on the expert projections, plus a fully trained router for the final checkpoints — alongside complete training state for reproducibility and crash-resume.

What's Inside

Every checkpoint folder contains:

• adapter_config.json + adapter_model.safetensors — LoRA (r=32, α=64) applied to the expert gate_up_proj and down_proj weights
• scheduler.pt + trainer_state.json — complete training state for seamless continuation
• gate.safetensors — the trained router (mlp.gate), learned in full precision (final checkpoints only)

<p align="center"> <img src="figures/training_loss.png" alt="Training loss and learning-rate schedule" width="90%"> </p> <p align="center"><em>Figure 1 — Training and validation loss across the full 1,000-step journey, with the cosine learning-rate schedule.</em></p> <p align="center"> <img src="figures/architecture.png" alt="Sparse MoE architecture" width="78%"> </p> <p align="center"><em>Figure 2 — The 8-expert, top-2 architecture that these LoRA adapters fine-tune.</em></p>

---

Usage

Attach the recommended final adapter to the upcycled base:

from transformers import AutoModelForCausalLM
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained(
    "pinkelephantlimited/pinkelephant-llm-48b-moe-base",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(
    model,
    "pinkelephantlimited/pinkelephant-llm-48b-adapters",
    subfolder="adapters/final",
)

Prefer a ready-to-run model? The trained adapters and router are already merged into the standalone pinkelephant-llm-48b release — no PEFT setup required.

---

Training Details

• Base: pinkelephant-llm-48b-moe-base (upcycled from pink-elephant-llm-14b)
• Data: ~100,000 code + math instruction examples
• Optimization: LoRA r=32, α=64 on expert projections; router trained in full precision
• Schedule: AdamW, cosine LR (peak 1.5e-4), 1,000 steps

---

Release Family

---

User Manual

Full family user manual: USER_MANUAL.pdf — what / which / where / how for every public artifact across the whole family, including how to load any step adapter and rebuild the loss curves.

---

Pink Elephant Limited — publishing large, sovereign language models.