pink-elephant-llm-14b

Pink Elephant LLM 14B

A 14.7B-parameter dense LLM by Pink Elephant Limited — the foundation of the Pink Elephant family — fine-tuned for code generation and mathematical reasoning.

Pink Elephant LLM 14B is a dense, decoder-only large language model released in bfloat16. It is the first and foundational member of the Pink Elephant lineage. From this dense core, Pink Elephant Limited upcycled an 8-expert sparse MoE and refined it through two fine-tuning epochs to build the rest of the family. It excels at producing correctly-executed code and cleanly derived mathematical answers from conversational prompts.

Released under the permissive MIT License — free for commercial use and modification, no attribution required.

---

Model Card

Parameter counts are verified directly from the shard header of final/model.safetensors (14,659,507,200 bf16 weights).

---

Benchmark Charts

---

Training Configuration

Final training loss ≈ 0.47 (cosine schedule decayed to ~0 LR), a converged, stable run with no observed divergence.

---

Training Data

Fine-tuned on two public, permissively-licensed instruction/solution corpora, then de-duplicated and shuffled with a fixed seed:

---

Evaluations (verified from humaneval_result.json, math500_result.json)

Both benchmarks measured on the merged bf16 model (final/) with greedy decoding.

---

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pinkelephantlimited/pink-elephant-llm-14b"
tok = AutoTokenizer.from_pretrained(model_id, subfolder="final", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, subfolder="final", torch_dtype=torch.bfloat16,
    device_map="cuda", trust_remote_code=True,
)

chat = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
prompt = tok.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**ids, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

Expected runtime on a 96 GB-class GPU: ≈ 55–65 decoded tokens/s.

---

Part of a Larger Story

This 14B core is where the Pink Elephant lineage begins. From this dense foundation, Pink Elephant Limited upcycled an 8-expert Mixture-of-Experts network and refined it through two supervised-fine-tuning epochs:

• pinkelephant-llm-48b — the first-epoch sparse flagship (47.7B MoE, ~14.7B active per token)
• pinkelephant-llm-48b-s — the refined two-epoch flagship, with training loss cut 32% (0.0616 → 0.0419)

One lineage, three milestones: a proven dense core, upcycled to sparse scale, refined twice.

---

Documentation & Reproducibility

• Training notebook: Max_Quality_LLM_96GB_VRAM.ipynb
• Technical report: TECHNICAL_REPORT.md
• Release report: RELEASE_REPORT.pdf
• Full family user manual: USER_MANUAL.pdf — what / which / where / how for every public artifact across the whole family

---

License & Commercial Use

MIT License (see LICENSE). Free for commercial and research use, modification, and redistribution, with or without attribution. Use at your own risk.

---

Pink Elephant Limited — Pink Elephant LLM 14B · 2026