Downloads · 30 days
0
pinkelephantlimited/pinkelephant-llm-48b-s-sft
pinkelephant-llm-48b-s-sft is a text generation model from pinkelephantlimited. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
100 full-precision LoRA adapters from a 100,000-step supervised fine-tuning campaign · published checkpoint by checkpoint
Downloads · 30 days
0
Access
Public
Updated Aug 21, 2026
Repo size
164 GB
Likes
0
Public
Click a slice to open those files.
.pt164 GB · 100%
From the Hugging Face model README
100 full-precision LoRA adapters from a 100,000-step supervised fine-tuning campaign · published checkpoint by checkpoint
<div align="center">
This repository is the training treasury of the Pink Elephant SFT program — the second act in our open alignment pipeline, after the 48B-S release and before Direct Preference Optimization (DPO).
At its heart are 100 full-precision LoRA adapters — the complete learning journey of a 47.7-billion-parameter sparse intelligence engine, captured at 1,000-step resolution from step 1,000 to step 100,000. Every adapter is standalone, loadable, and evaluable.
The campaign was run with an industrial-standard QLoRA recipe — the same cost-efficient methodology Meta and the open-weight ecosystem use to fine-tune flagship-scale models. The frozen base is stored in 4-bit NF4 purely as the memory-efficient training vehicle; all computation ran in native bf16, and the published adapters themselves are full precision. This is the efficient way large models get fine-tuned — and it is exactly why the whole 47.7B model was fine-tuned on a single GPU.
The story is measured, not marketed. Training loss fell from 0.65 → ~0.44 and held-out evaluation identified step 90,000 as the best adapter (CE 0.5392) — the checkpoint we recommend for release. Every number here is real, logged, and reproducible from the public artifacts in this repository.
This is the treasury before the crown. The SFT stage is complete; DPO is next on the road map. Investors, auditors, and engineers can watch — and reproduce — exactly how a flagship model learns to follow instructions, step by step.
| Asset | What it is | Value |
|---|---|---|
| 100 full-precision adapters | sft/sft-lora-step1000.pt … sft/sft-lora-step100000.pt | The entire training journey at 1,000-step resolution — a living time-lapse of learning |
| The training vehicle | The 47.7B MoE network in 4-bit NF4, compute in bf16 | Whole-model fine-tuning on a single 96 GB GPU — cost-efficient, industry standard |
| Full campaign record | training/sft/ — report, eval results, manifest, notebooks | Auditable from first step to release decision |
| Reproducibility kit | pinned requirements.txt, SETUP_GUIDE.md, corrected notebook | Anyone can re-run the campaign and verify the numbers |
| MIT license | — | Free for commercial use, modification, and redistribution |
| Specification | Value |
|---|---|
| Base model | Pink Elephant 48B-S — MoE, 8 experts, top-2, 47.69B parameters |
| Method | QLoRA — frozen base in 4-bit NF4 (double-quantized) for storage, all training compute in bf16 |
| Adapters | Full precision — LoRA weights stored natively, mergeable into the bf16 base |
| LoRA rank / alpha / dropout | 32 / 64 / 0.05 |
| LoRA target modules | qkv_proj, o_proj, gate_up_proj, down_proj |
| Optimizer | PagedAdamW8bit, lr 2e-4, cosine, 50-step warm-up |
| Sequence length / batch | 1,024 tokens · micro-batch 4 · grad-accum 8 (effective 32/step) |
| Dataset | OpenHermes-2.5 — 1,001,551 tokenized instructions (ChatML, 1,024 cap) |
| Training steps | 100,000 |
| Adapters published | 100 (every 1,000 steps) |
| Hardware | Single NVIDIA RTX PRO 6000 Blackwell, 96 GB VRAM (~28–30 GB resident) |
| Final training loss | ~0.44 at step 88,999 (near-plateau) |
| Held-out best | step 90,000 — 0.5392 CE |
| Recommended release | sft/sft-lora-step90000.pt |
| License | MIT — free for commercial use |
Training loss fell steadily from 0.65 in the early campaign to a plateau near 0.44 — a healthy, stable SFT convergence over 100,000 steps. The single high point at step 69,019 is a documented resume artifact / bad batch, not a divergence; the loss immediately returned to the falling trend.
After training, three released checkpoints were measured on a clean held-out set (2,000 sampled sequences the model had never seen):
<p align="center"> <img src="https://huggingface.co/pinkelephantlimited/pinkelephant-llm-48b-s-sft/resolve/main/figures/fig_heldout.png" alt="Held-out evaluation across release checkpoints" width="88%"> </p>| Checkpoint | Held-out CE loss |
|---|---|
| step 80,000 | 0.5424 |
| step 90,000 | 0.5392 ← best |
| step 100,000 | 0.5578 |
Held-out loss fell from step 80k → 90k, then rose at step 100k (+0.0185) — the classic signature of a model beginning to overfit its training distribution in the tail. This empirically confirms the early-stop at ~100k decision. There is no justification for another 150,000 steps; the released adapter is step 90,000.
Every 1,000 steps produced a full-precision adapter, so the model's growth can be watched — and evaluated — at any moment of its life:
sft/
├── sft-lora-step1000.pt ← the first checkpoint
├── sft-lora-step2000.pt
├── ...
├── sft-lora-step90000.pt ← best held-out (0.5392) — recommended release
├── ...
└── sft-lora-step100000.pt ← the end of the campaign
Each adapter is standalone (weights + config) and loads onto the base model, which ships in native bf16 from the flagship repository. 100 adapters. One continuous story. This is what transparent, industrial-standard model development looks like.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, set_peft_model_state_dict
from huggingface_hub import hf_hub_download
REPO = "pinkelephantlimited/pinkelephant-llm-48b-s-sft"
BASE = "pinkelephantlimited/pinkelephant-llm-48b-s"
tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(
BASE, torch_dtype=torch.bfloat16,
trust_remote_code=True, device_map="cuda", # or load_in_4bit=True for a smaller footprint
)
model = get_peft_model(model, LoraConfig(
r=32, lora_alpha=64,
target_modules=["qkv_proj", "o_proj", "gate_up_proj", "down_proj"],
lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
))
# load any published adapter — e.g. the recommended release
sd = torch.load(hf_hub_download(repo_id=REPO, filename="sft/sft-lora-step90000.pt",
repo_type="model"), map_location="cpu", weights_only=False)
set_peft_model_state_dict(model, sd["adapter"]) # ← the correct PEFT loader
Note for engineers: adapters must be loaded with
set_peft_model_state_dict, notmodel.load_state_dict(..., strict=False). A subtle key mismatch (lora_A.weightvslora_A.default.weight) can silently load nothing. This was discovered, fixed, and documented during our own audit — seetraining/sft/SETUP_GUIDE.md.
chat = [{"role": "user", "content": "Explain what a Mixture of Experts model is in one sentence."}]
prompt = tok.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
out = model.generate(prompt.to(model.device), max_new_tokens=256, do_sample=True, temperature=0.7,
eos_token_id=[tok.eos_token_id, tok.convert_tokens_to_ids("<|im_end|>")])
print(tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True))
from peft import PeftModel
model = PeftModel.from_pretrained(model, hf_hub_download(repo_id=REPO, filename="sft/sft-lora-step90000.pt"))
merged = model.merge_and_unload()
merged.save_pretrained("./pink-elephant-48b-s-sft-merged") # a single bf16 model, no adapters
Everything needed to reproduce the training is public in training/sft/:
| Artifact | Contents |
|---|---|
requirements.txt | Pinned, verified environment (transformers 5.14.1, bnb 0.50.1, peft 0.20.0, accelerate 1.14.0) |
SETUP_GUIDE.md | Full GPU-workstation recovery procedure — validated from scratch |
sft_notebook_cells.json | The corrected, complete training notebook (incl. torch.library patch + loss persistence) |
SFT_REPORT.md | The training report in markdown |
heldout_eval.json | Machine-readable held-out evaluation results |
RELEASE.json | The release manifest (adapter = step 90,000) |
One hard-won lesson, documented publicly: transformers must stay on 5.14.1. Version 5.15.x changed gradient-checkpointing defaults and crashes immediately with bitsandbytes 4-bit training. The pinned environment reproduces the campaign exactly.
Fine-tuning a 47.7B MoE flagship is a compute-and-memory problem. QLoRA answers it the industry-standard way:
For full-precision deployment, the base model ships natively in bf16 in the flagship
repository (pinkelephant-llm-48b-s, verified end-to-end on Blackwell at 96 GB residency).
The whole 100,000-step campaign ran on a single GPU — an engineering fact that separates a laboratory from a production discipline:
| Efficiency metric | Value |
|---|---|
| Model fine-tuned | 47.7B parameters (8-expert MoE) |
| Peak training VRAM | ~28–30 GB on an RTX PRO 6000 Blackwell (96 GB) |
| Memory efficiency vs. bf16 fine-tune | ~3.4× lower peak VRAM |
| Full campaign | 100,000 steps, 1,001,551 instructions, single GPU |
| Checkpoint cadence | every 1,000 steps, pushed to Hugging Face live |
This is fine-tuning at laboratory scale on a single workstation — the same campaign that would otherwise demand a multi-GPU cluster. Every checkpoint, every number, every log is public. That is what disciplined, transparent, investor-auditable engineering looks like.
This repository is stage two of a four-stage industrial alignment pipeline:
pinkelephant-llm-48b-s)The SFT stage is done, documented, and reproducible. DPO reuses the same QLoRA infrastructure. The treasure grows.
| Repository | Role in the Story |
|---|---|
pinkelephant-llm-48b-s | The refined flagship — two epochs, merged, native bf16, ready to run |
pinkelephant-llm-48b-s-sft (this repo) | The SFT adapter treasury — 100 full-precision LoRA adapters + the efficient training vehicle |
pinkelephant-llm-48b | The first-epoch flagship |
pinkelephant-llm-48b-moe-base | The upcycled sparse base — where the experts were born |
pinkelephant-llm-48b-adapters | LoRA adapters + trained router, published every 100 steps |
pink-elephant-llm-14b | The dense 14B core — where it all began |
pinkelephant-llm-48b-sft-openhermes-1m | The tokenized instruction dataset behind the SFT campaign |
| Document | Location |
|---|---|
| SFT Training Report (PDF) | SFT_REPORT.pdf |
| SFT Training Report (markdown) | training/sft/SFT_REPORT.md |
| Setup & recovery guide | training/sft/SETUP_GUIDE.md |
| Release manifest | training/sft/RELEASE.json |
| Held-out evaluation | training/sft/heldout_eval.json |
Flagship documentation (in pinkelephantlimited/pinkelephant-llm-48b-s):
| Document | Contents |
|---|---|
| Release Report | The 48B-S release narrative — architecture, lineage, training, quality |
| Verification Test Report | Instrumented end-to-end test + benchmarks on real Blackwell hardware |
| User Manual | What / which / where / how for every public artifact in the family |
training/sft/).SETUP_GUIDE.md)
was found during our own audit and published with the fix. Transparency is the brand.SETUP_GUIDE.md). The held-out numbers are per-adapter
quality, which is exactly how the release decision was made.Released under the MIT License — free for commercial and research use, modification, and redistribution, with or without attribution.
Pink Elephant Limited — publishing large, sovereign language models.
Built by Pink Elephant Limited · 2026