Downloads · 30 days
0
pinkelephantlimited/pinkelephant-llm-48b-s-dpo
pinkelephant-llm-48b-s-dpo is a text generation model from pinkelephantlimited. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
49 full-precision LoRA adapters from an alignment campaign that found its turning point — Direct Preference Optimization, published checkpoint by checkpoint
Downloads · 30 days
0
Access
Public
Updated Aug 21, 2026
Repo size
101 GB
Likes
0
Public
Click a slice to open those files.
.pt94.4 GB · 100%
From the Hugging Face model README
49 full-precision LoRA adapters from an alignment campaign that found its turning point — Direct Preference Optimization, published checkpoint by checkpoint
<div align="center">
This repository is the alignment treasury of the Pink Elephant alignment program — the third act in our open pipeline, after the 48B-S release and the Supervised Fine-Tuning (SFT) adapter treasury. This is where the flagship learns what humans prefer, not just how to follow instructions.
At its heart are 49 full-precision LoRA adapters — the complete alignment journey,
captured at 250-step resolution from step 250 to step 12,250, plus the single
dpo-best.pt adapter at the measured min-loss step 11,579. Every adapter is standalone,
loadable, and evaluable.
The campaign was run with an industrial-standard QLoRA-for-DPO recipe: a frozen reference model alongside a trainable policy branch, minimizing a preference loss that encourages the model to favor chosen completions over rejected ones while staying close to the reference — the modern, direct alternative to reward-model + RLHF pipelines.
The defining engineering decision of this run is the turning-point early stop, made on evidence: we did not stop at an arbitrary step ceiling. We watched the loss curve, found its true minimum of 0.4763 at step 11,579, then stopped 1,500 steps later once it was clear the loss had turned. The released adapter is the min-loss step 11,579. On a clean 2,000-pair held-out preference set, this adapter reaches ~0.65 preference accuracy — a meaningful, honest alignment signal well above chance, measured and reproducible from the public artifacts here.
This is where the flagship learns to prefer what it was taught. SFT taught it how; DPO teaches it what. Investors, auditors, and engineers can watch — and reproduce — exactly how a flagship model's preferences are shaped, checkpoint by checkpoint.
| Asset | What it is | Value |
|---|---|---|
| 49 full-precision adapters | dpo/dpo-lora-step250.pt … dpo/dpo-lora-step12250.pt + dpo/dpo-best.pt | The entire alignment journey at 250-step resolution — a living time-lapse of preference learning |
| The training vehicle | The 47.7B MoE network in 4-bit NF4, compute in bf16, policy + frozen reference branches | Whole-model DPO on a single 96 GB GPU — cost-efficient, industry standard |
| The turning-point decision | Auto-detected loss minimum at step 11,579 (0.4763) | Early stop on evidence, not an arbitrary ceiling |
| Full campaign record | training/dpo/ — report, eval results, manifest, notebooks, runbook | Auditable from first step to release decision |
| Reproducibility kit | pinned requirements.txt, SETUP_GUIDE.md, DPO_RUNBOOK.md, corrected notebooks | Anyone can re-run the campaign and verify the numbers |
| MIT license | — | Free for commercial use, modification, and redistribution |
| Specification | Value |
|---|---|
| Base model | Pink Elephant 48B-S — MoE, 8 experts, top-2, 47.69B parameters |
| SFT warm start | pink-elephant-llm-48b-s-sft/sft/sft-lora-step90000.pt loaded into both policy and reference |
| Method | QLoRA-for-DPO — frozen base in 4-bit NF4 (double-quantized), all training compute in bf16 |
| Adapters | Full precision — LoRA weights stored natively, mergeable into the bf16 base |
| LoRA rank / alpha / dropout | 32 / 64 / 0.05 |
| LoRA target modules | qkv_proj, o_proj, gate_up_proj, down_proj |
| Optimizer | PagedAdamW8bit, lr 1e-5 |
| DPO objective | β = 0.1 preference loss against a frozen reference |
| Sequence length / batch | 1,024 tokens · micro-batch 2 · grad-accum 16 (effective 32/step) |
| Dataset | OpenHermes-2.5-preferences-v0-deduped — N_TRAIN 120,000 / N_HELDOUT 2,000 preference pairs |
| Campaign steps | 13,079 (resumed from step ~10,000, then turning-point early stop) |
| Turning point (min loss) | step 11,579 — 0.4763 |
| Adapters published | 49 (every 250 steps) + dpo-best.pt at the min-loss step |
| Hardware | Single NVIDIA RTX PRO 6000 Blackwell, 96 GB VRAM |
| Min-loss training loss | 0.4763 at step 11,579 |
| Held-out best | step 11,579 — preference accuracy ~0.65 (2,000 held-out pairs) |
| Recommended release | dpo/dpo-best.pt (min-loss step 11,579) |
| License | MIT — free for commercial use |
Training loss descended toward its global minimum of 0.4763 at step 11,579, then turned and began to rise again. We did not chase more steps past the turn. The turning-point early-stop fired 1,500 steps later (step 13,079) and correctly terminated the campaign so that the released adapter is the true minimum. That is DPO alignment stopped on evidence, not on a calendar.
After training, five checkpoints spanning the turning point were measured on a clean held-out set of 2,000 preference pairs the model had never seen:
<p align="center"> <img src="https://huggingface.co/pinkelephantlimited/pinkelephant-llm-48b-s-dpo/resolve/main/figures/fig_heldout.png" alt="Held-out preference accuracy across release checkpoints" width="88%"> </p>| Checkpoint | Held-out preference acc | Mean DPO loss |
|---|---|---|
| step 11,000 | 0.6525 | 0.6282 |
| step 11,500 | 0.6480 | 0.6271 |
| step 11,579 (best) | 0.6470 | 0.6272 ← best |
| step 11,750 | 0.6535 | 0.6279 |
| step 12,000 | 0.6485 | 0.6315 |
Preference accuracy is stable around ~0.65 (well above the 0.50 chance floor) across all five checkpoints in the window — the model has genuinely and consistently learned the preference signal, and continued training past the turning point does not push held-out accuracy higher (it fluctuates within noise). This empirically confirms the early-stop at the min-loss step 11,579: the released adapter is the loss minimum, and holding it out against the reference-preference signal is exactly the right release point.
What ~0.65 means, honestly. A preference model is not expected to hit 1.00 — that would mean the model simply memorized the training preference pairs. A held-out accuracy of ~0.65 across unseen pairs, comfortably above chance, is the signature of a model that has internalized a generalizable preference signal rather than memorized examples. Combined with the min-loss turning point, step 11,579 is the principled release.
Every 250 steps produced a full-precision adapter, so the model's preference change can be watched — and evaluated — at any moment of its alignment:
dpo/
├── dpo-lora-step250.pt ← the first checkpoint
├── dpo-lora-step500.pt
├── ...
├── dpo/dpo-lora-step11500.pt
├── dpo-best.pt ← min-loss step 11,579 (0.4763) — recommended release
├── dpo/dpo-lora-step11750.pt
├── ...
└── dpo-lora-step12250.pt ← last published 250-step checkpoint
Each adapter is standalone (weights + config) and loads onto the base model with the same PEFT loader used across the family. 49 adapters + the best. One continuous alignment story, cut at its true turning point. This is what transparent, industrial-standard model alignment looks like.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, set_peft_model_state_dict
from huggingface_hub import hf_hub_download
REPO = "pinkelephantlimited/pinkelephant-llm-48b-s-dpo"
BASE = "pinkelephantlimited/pinkelephant-llm-48b-s"
tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(
BASE, torch_dtype=torch.bfloat16,
trust_remote_code=True, device_map="cuda", # or load_in_4bit=True for a smaller footprint
)
model = get_peft_model(model, LoraConfig(
r=32, lora_alpha=64,
target_modules=["qkv_proj", "o_proj", "gate_up_proj", "down_proj"],
lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
))
# load the recommended release — dpo-best.pt (min-loss step 11,579)
sd = torch.load(hf_hub_download(repo_id=REPO, filename="dpo/dpo-best.pt",
repo_type="model"), map_location="cpu", weights_only=False)
set_peft_model_state_dict(model, sd["adapter"]) # ← the correct PEFT loader
For engineers: as in the SFT treasury, adapters must be loaded with
set_peft_model_state_dict, notmodel.load_state_dict(..., strict=False)— a subtle key mismatch (lora_A.weightvslora_A.default.weight) can silently load nothing. This was discovered, fixed, and documented during the SFT audit and is carried into every downstream stage.
chat = [{"role": "user", "content": "Explain what a Mixture of Experts model is in one sentence."}]
prompt = tok.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
out = model.generate(prompt.to(model.device), max_new_tokens=256, do_sample=True, temperature=0.7,
eos_token_id=[tok.eos_token_id, tok.convert_tokens_to_ids("<|im_end|>")])
print(tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True))
from peft import PeftModel
model = PeftModel.from_pretrained(model, hf_hub_download(repo_id=REPO, filename="dpo/dpo-best.pt"))
merged = model.merge_and_unload()
merged.save_pretrained("./pink-elephant-48b-s-dpo-merged") # a single bf16 model, no adapters
Everything needed to reproduce the training is public in training/dpo/:
| Artifact | Contents |
|---|---|
requirements.txt | Pinned, verified environment (transformers, bnb, peft, accelerate — see below) |
SETUP_GUIDE.md | Full GPU-workstation recovery procedure — validated from scratch |
DPO_RUNBOOK.md | Step-by-step operation runbook for the DPO campaign on our Blackwell workstation |
notebook.py / notebook_v2.py | The DPO training notebooks (v2 adds the turning-point early stop) |
DPO_REPORT.md | The training/alignment report in markdown |
heldout_eval.json | Machine-readable held-out preference evaluation results |
RELEASE.json | The release manifest (adapter = dpo-best.pt, step 11,579) |
loss_history.json | Machine-readable training loss curve (the turning point is visible in it) |
tokenized_pairs.npz | Tokenized preference pairs (prompt / chosen / rejected) |
ref_logprobs.pt | Frozen reference-model log-probabilities used by the DPO loss |
One engineering reality, documented honestly: the campaign resumed from the exploratory 10k-ceiling run at step ~10,000. The turning-point logic in
notebook_v2.pyran from there to step 13,079, where it correctly early-stopped at the min-loss step 11,579. The loss curve published here is the decisive phase: the descent to the minimum, the turn, and the evidence-based stop.
Aligning a 47.7B MoE flagship is a compute-and-memory problem, made harder by DPO's need for a frozen reference alongside the trainable policy. QLoRA-for-DPO answers it the industry-standard way:
ref_logprobs.pt).For full-precision deployment, the base model ships natively in bf16 in the flagship
repository (pinkelephant-llm-48b-s).
The whole alignment campaign ran on a single GPU — the same discipline that carried the SFT stage:
| Efficiency metric | Value |
|---|---|
| Model aligned | 47.7B parameters (8-expert MoE) + a frozen reference branch |
| Peak training VRAM | ~55–60 GB on an RTX PRO 6000 Blackwell (96 GB) — two 4-bit copies resident |
| Alignment campaign | 13,079 steps (→ min-loss 11,579), 120,000 preference pairs, single GPU |
| Checkpoint cadence | every 250 steps, pushed to Hugging Face live |
| Early stop | turning point detected on the loss curve, not an arbitrary ceiling |
This is preference alignment at laboratory scale on a single workstation. Every checkpoint, every number, every log is public.
This repository is stage three of a five-stage industrial alignment pipeline:
pinkelephant-llm-48b-s)The DPO stage is done, documented, and reproducible — stopped at its true minimum, not an arbitrary ceiling. SFT taught the flagship how; DPO taught it what; verification proves it.
This is the verification stage (pipeline step 4): the fully-aligned model — base 48B-S +
best SFT adapter (step 90000) + min-loss DPO adapter (dpo-best, step 11,579) — fused in
bf16 and evaluated on an NVIDIA RTX PRO 6000 Blackwell. These are the first public,
reproducible numbers for the aligned flagship, closing the open question raised in this
card's own limitations.
| Benchmark | Aligned 48B-S | Protocol |
|---|---|---|
| GSM8K (math word problems) | 68.5% (61/89) | Random sample of GSM8K test · answer is extraction |
| MMLU (knowledge, 5-shot) | 60.0% (24/40) | 40 balanced questions across 7 subjects · \boxed{letter} extraction |
Greedy decoding. Extraction is robust (
answer isrefex for GSM8K,\boxed{letter}after the CoT for MMLU); the model is a reasoning model, so a naive "last token" metric under-reports and is not used.
Everything is reproducible. All verification tooling — the eval harness
(run_benchmarks.py), the adapter-stack converters (adapter_to_peft.py), the full-pipeline
notebook (verify_full_model.py), the structural checkers, the saved per-question results,
and the formal report — lives in the dedicated verification repo:
pinkelephantlimited/pinkelephant-llm-48b-s-verification.
The verification repo covers the whole final model (base 48B-S + SFT step90000 + DPO
dpo-best), not just the DPO stage. Any future run can download those tools instead of
rewriting them — reuse, no rewrite.
| Repository | Role in the Story |
|---|---|
pinkelephant-llm-48b-s | The refined flagship — two epochs, merged, native bf16, ready to run |
pinkelephant-llm-48b-s-sft | The SFT adapter treasury — 100 full-precision LoRA adapters (step 90,000 released) |
pinkelephant-llm-48b-s-dpo (this repo) | The DPO adapter treasury — 49 full-precision LoRA adapters + the min-loss dpo-best.pt, stopped at the turning point |
pinkelephant-llm-48b | The first-epoch flagship |
pinkelephant-llm-48b-moe-base | The upcycled sparse base — where the experts were born |
pinkelephant-llm-48b-adapters | LoRA adapters + trained router, published every 100 steps |
pink-elephant-llm-14b | The dense 14B core — where it all began |
pinkelephant-llm-48b-sft-openhermes-1m | The tokenized instruction dataset behind the SFT campaign |
| Document | Location |
|---|---|
| DPO Training Report (PDF) | DPO_REPORT.pdf |
| DPO Training Report (markdown) | training/dpo/DPO_REPORT.md |
| Setup & recovery guide | training/dpo/SETUP_GUIDE.md |
| DPO operation runbook | training/dpo/DPO_RUNBOOK.md |
| Release manifest | training/dpo/RELEASE.json |
| Held-out evaluation | training/dpo/heldout_eval.json |
| Training loss curve | training/dpo/loss_history.json |
Flagship documentation (in pinkelephantlimited/pinkelephant-llm-48b-s):
| Document | Contents |
|---|---|
| Release Report | The 48B-S release narrative — architecture, lineage, training, quality |
| Verification Test Report | Instrumented end-to-end test + benchmarks on real Blackwell hardware |
| User Manual | What / which / where / how for every public artifact in the family |
dpo-best.pt at the min-loss step 11,579.training/dpo/).Released under the MIT License — free for commercial and research use, modification, and redistribution, with or without attribution.
Pink Elephant Limited — publishing large, sovereign language models.
Built by Pink Elephant Limited · 2026