PLLuM-12B-chat-2512-EXL3

<p align="center"> <img src="PLLuM_logo_RGB_color.png" width="400"> </p>

🇵🇱 PLLuM-12B-chat-2512 — EXL3 4.5bpw

EXL3 quantization of CYFRAGOVPL/PLLuM-12B-chat-2512 — a 12B Polish chat LLM by the PLLuM / HIVE AI consortium, optimized for 12 GB VRAM.

Quantized by @Mati83moni using ExLlamaV3 (v0.0.43) on AWS EC2.

Created on request from the HuggingFace community (Discussion #1).

---

⚠️ Note on HuggingFace "Model size" Display

You may notice the sidebar shows "Model size: 4B params" — this is not accurate and is a known limitation of HuggingFace's automatic parameter counter for EXL3 models.

This model has the full ~12B parameters of the original PLLuM-12B-chat-2512.

Why the discrepancy? EXL3 uses trellis-encoded weight packing, where each weight tensor is compressed into a .trellis sub-tensor with a different shape than the original weight matrix (e.g. a [14336, 5120] weight becomes a [320, 896, 64] int16 trellis tensor). HuggingFace's parameter counter sums raw tensor element counts from the safetensors header, which — for trellis-packed formats — does not correspond to the actual number of model parameters. This affects all EXL3 models on HuggingFace, not just this one.

Ground truth parameter count: 12B (verify via config.json: hidden_size=5120, num_hidden_layers=40, intermediate_size=14336, vocab_size=131072 — matching the original PLLuM-12B-chat-2512 architecture exactly).

---

📦 Quantization Specification

quantization_config.json stores "bits": 4 (an integer) — this is how ExLlamaV3 v0.0.43 records the base bitrate; the actual per-tensor allocation below averages to 4.5 bpw over the decoder linear layers.

Quantization Strategy (auto)

Per-tensor bitrates read from quantization_config.json (tensor_storage[*].bits_per_weight):

---

💾 VRAM Requirements

Measured on NVIDIA Tesla T4 (16 GB), ExLlamaV3 0.0.43 (+ sm_75 patch, see Benchmarks), batch size 1, FP16 KV cache, model and cache on one GPU. "nvidia-smi" includes the CUDA context; "PyTorch" is torch.cuda.max_memory_allocated.

The "Fits 12 GB?" column is derived from the T4 measurement (peak < 95 % of 12 GiB = ✅, < 12 GiB = ⚠️). Peak VRAM on other GPUs/drivers can differ slightly (CUDA context size, attention backend). The FP16 KV cache costs 160 KiB per token (40 layers × 8 KV heads × 128 dims × 2 × FP16).

---

🏗️ Model Architecture

---

🇵🇱 About PLLuM

PLLuM (Polish Large Language Model) is a family of LLMs specialized for the Polish language, developed by the PLLuM consortium and continued under HIVE AI — an alliance of research institutions delivering digital public services for Polish public administration.

Key Features

• 🏆 State-of-the-art in Polish NLP tasks and Polish public administration benchmarks
• 📚 Largest organic Polish instruction dataset — manually authored by human experts, not distilled
• 🛡️ Polish preference corpus — first of its kind, with demographically diverse annotators
• 🔬 Instruction-tuned via SFT + preference learning + advanced alignment
• 🌐 Bilingual — Polish primary + English for broader generalization

---

🚀 How to Use

With TabbyAPI

# config.yml
model_dir: /path/to/PLLuM-12B-chat-2512-EXL3-4.5bpw
max_seq_len: 4096

python start.py --model-dir PLLuM-12B-chat-2512-EXL3-4.5bpw

With ExLlamaV3 (Python)

from exllamav3 import ExLlamaV3Config, ExLlamaV3, ExLlamaV3Tokenizer
from exllamav3.generator import ExLlamaV3DynamicGenerator

model_dir = "PLLuM-12B-chat-2512-EXL3-4.5bpw"

config = ExLlamaV3Config(model_dir)
model = ExLlamaV3(config)
model.load()

tokenizer = ExLlamaV3Tokenizer(config)
generator = ExLlamaV3DynamicGenerator(model=model, tokenizer=tokenizer)

prompt = "[INST]Wyjaśnij czym jest uczenie maszynowe w 3 zdaniach.[/INST]"
output = generator.generate(prompt=prompt, max_new_tokens=512)
print(output)

---

💬 Chat Template

This model uses the Mistral Instruct template exactly as defined in chat_template.jinja (no spaces around the tags):

<s>[INST]<system prompt (optional)>\n\n<user message>[/INST]<assistant response></s>[INST]<next user message>[/INST]

The chat_template.jinja file is included in this repo and will be automatically detected by TabbyAPI and ExLlamaV3.

Recommended Sampling Parameters

---

💻 Hardware Requirements

Minimum (12 GB VRAM)

• GPU: Any GPU with ≥12 GB VRAM for contexts up to 16K (measured peak on T4: see VRAM Requirements)
• RAM: 16 GB system RAM
• Storage: 8 GB free
• Software: ExLlamaV3 v0.0.43+ or TabbyAPI with exllamav3 backend. Official ExLlamaV3 wheels target Ampere (sm_80) and newer; Turing GPUs (T4, RTX 20xx) need the compatibility patch described under Reproduce.

Measured speed

Only one GPU was tested. Speeds for other GPUs are not measured and are intentionally not listed.

T4 numbers are a lower bound: on sm_75 the patched kernels replace cp.async with synchronous copies and split each tensor-core MMA in two, and FlashAttention-2 is unavailable (Triton paged attention is used for decoding). They are not representative of Ampere/Ada/Hopper GPUs.

💡 Why EXL3? EXL3 is the native format of ExLlamaV3, an inference library for local GPUs. This repository makes no speed comparison with other quantization formats; the only speed numbers here are the T4 measurements above.

---

📊 Benchmarks

All numbers in this section were measured on 2026-10-04 on NVIDIA Tesla T4 16 GB (15360 MiB usable, sm_75), driver 580.178.04, Kaggle, with ExLlamaV3 0.0.43 (commit c5d9c65) + sm_75 compatibility patch, PyTorch 2.11.0+cu128 (CUDA 12.8), Python 3.13.15, transformers 5.16.1, datasets 4.8.5, Triton 3.6.0, lm-evaluation-harness 0.4.13. Model revisions: this repo 278a520af92f, original CYFRAGOVPL/PLLuM-12B-chat-2512 19a26cf61648. Seed 0 everywhere a seed applies (all decoding is greedy). All measured values are in benchmarks/results.json; the scripts are in benchmarks/ (see Reproduce).

Why a patch? ExLlamaV3 v0.0.43 is built for sm_80+ only; the free Kaggle GPUs are Tesla T4 (sm_75). A minimal patch (benchmarks/patch_sm75.py, 7 items, diff ~100 lines) makes it run on Turing. On sm_80+ the patched sources compile to the upstream device code. The patched kernels were validated: for all 281 linear layers the fused EXL3 GEMM kernel and the reconstruct-then-cuBLAS path agree to a worst-case relative error of 5.5e-04 (FP16 rounding level) for 1, 4 and 16 rows.

1. Perplexity — EXL3 vs original (context 2048)

• Method: upstream eval/model_diff.py (ExLlamaV3 v0.0.43). Model A = this EXL3 repo, model B = the original checkpoint. Both models process the same token ids module by module (layer-by-layer, so the 24.5 GB original fits on a 16 GB T4); 100 non-overlapping rows of 2048 tokens (stride 2048), no BOS, PPL over all 2047 next-token predictions per row.
• "Original (BF16 weights)" = the BF16 checkpoint run by ExLlamaV3 with FP16 activations (T4 has no BF16 tensor cores); the result comes from model_diff.py, not from an external source.
• Upstream eval/ppl.py could not be used as an independent cross-check: in v0.0.43 it crashes in model_init.init ('Namespace' object has no attribute 'mtp').

Polish evaluation text. wikimedia/wikipedia, config 20231101.pl, split train, dataset revision b04c8d1ceb2f: the first 60 articles in dataset order with ≥ 3,000 characters, joined with blank lines (780,272 characters, 248,098 tokens; SHA-256 0c54af15dd07d429…); the first 100 × 2048 tokens are used. First articles: AWK, ASCII, Atom, Aksjomat, Arytmetyka. Overlap with ExLlamaV3's default calibration corpus (13-word shingles): Polish text 0 / 105066 shingles, WikiText-2 test 0 / 206132. Wikipedia is very likely part of PLLuM's own training data, which lowers the absolute PPL of both models equally; only the EXL3-vs-original difference is meaningful.

2. KL divergence and top-K agreement vs original (context 2048)

Same runs as the perplexity table (model_diff.py, 100 × 2048 tokens per language). KL is computed over the full 131,072-token vocabulary and averaged per position. "Top-K agreement" = fraction of positions where the K most likely tokens of both models are identical (in order); "Top-1 acc." = fraction of positions where the model's argmax equals the actual next token.

3. Polish downstream tasks (0-shot, EXL3 vs original)

• Harness: lm-evaluation-harness 0.4.13 (simple_evaluate, num_fewshot=0, no chat template, BOS prepended, max length 2048, limit=None = full test sets), with an ExLlamaV3 model wrapper (benchmarks/lm_exl3.py): log-likelihoods from one full forward pass per sequence; generative tasks (PolEmo2) use greedy decoding with the task's stop strings. Values are percentages ± bootstrap/standard error as reported by lm-eval.
• Original BF16: the BF16 checkpoint run by the same ExLlamaV3 build and the same harness/wrapper/settings, as FP16 with layers split over 2× Tesla T4 (≈ 30 GB; it does not fit one 16 GB card). The PLLuM model card and arXiv:2511.03823 publish no numbers for the -2512 checkpoints on these tasks.
• Δ = EXL3 − original in percentage points. With n = 100–1172 the standard errors are 1.3–4.4 pp, so differences smaller than about two standard errors are not statistically meaningful.

4. Performance and VRAM (Tesla T4, batch 1)

• Synthetic random token ids (seed 0); prefill in chunks of 2048 into an FP16 KV cache of exactly the listed size; decode = 128 greedy single-token forward passes at the end of the filled cache. 1 warm-up repetition discarded, median of the next 3. VRAM: nvidia-smi sampled every 50 ms (includes CUDA context) and PyTorch allocator peak.
• Decode at the start of the context (tokens 1–129, 2048-token cache): 14.0 tok/s (separate run; repeated 2048-context run in that session: prefill 349 tok/s, decode 7.7 tok/s). The drop with context length comes from the Triton paged-attention fallback used on Turing (FlashAttention-2 is unavailable on sm_75).
• Upstream eval/perf.py could not be used: in v0.0.43 it crashes in model_init.init ('Namespace' object has no attribute 'mtp').

5. Qualitative check (10 fixed Polish prompts)

10 fixed prompts (diacritics, grammar, inflection, summarization, instructions), greedy decoding, max_new_tokens=512, chat template, EXL3 model. Full prompts and answers: benchmarks/qualitative_pl.json. Observations (manual reading, not a metric): fluent Polish with correct diacritics in all answers; correct inflection table for przyjaciel; correct numeral agreement; good one/two-sentence summaries. Errors seen: prompt 1 (restore diacritics) changed words instead of only adding diacritics; prompt 2 produced two sentences instead of three; prompt 4 gave a wrong justification; prompt 9 contains factual inaccuracies about the ID-card procedure; prompt 10 invents a detail (a bad smell) not in the request. Same prompts on the original BF16 model (2× T4, same settings): answers 1 and 4 are token-for-token identical to EXL3 (the same errors), the other 8 differ in wording only; BF16 shows the same kinds of errors (prompt 2: one sentence; prompt 9: similar procedural inaccuracies; prompt 10: invents a floor number). The observed errors are therefore behaviour of the original model, not quantization artefacts. Both answer sets: benchmarks/qualitative_pl.json (EXL3) and benchmarks/qualitative_pl_bf16.json (BF16).

---

📊 Quality Assessment

Measured against the original weights on identical token sequences (ctx 2048): perplexity changes by +1.15 % on Polish Wikipedia and +0.88 % on English WikiText-2; mean KL divergence KL(BF16‖EXL3) is 0.0193 (PL) / 0.0172 (EN) nats per token, and the EXL3 model picks the same top-1 token as the original in 94.9 % (PL) / 94.5 % (EN) of positions. On six Polish downstream tasks the EXL3 model scores within one standard error of the original (differences between −1.3 and +1.0 pp on the primary metrics). This is a measurement on two text sets at one context length, not a guarantee for every use case.

---

🔁 Reproduce

All scripts used for the numbers on this card are in benchmarks/; benchmarks/README.md lists every file, the library versions and the GPU requirements. All measured values are in benchmarks/results.json. The evaluation texts are rebuilt from public datasets by make_eval_texts.py, which checks their SHA-256. On an Ampere or newer GPU skip the patch/build in step 0 and install the official wheel.

# 0) ExLlamaV3 v0.0.43 for Turing (T4): apply the patch, build for sm_75 (~10 min on 4 vCPU)
git clone --depth 1 --branch v0.0.43 https://github.com/turboderp-org/exllamav3
python benchmarks/patch_sm75.py exllamav3                      # idempotent
(cd exllamav3 && TORCH_CUDA_ARCH_LIST=7.5 MAX_JOBS=4 pip wheel --no-deps --no-build-isolation -w ../wheels .)
pip install --no-deps wheels/exllamav3-0.0.43-*.whl
pip install rich pyyaml marisa_trie pydantic ninja "lm-eval==0.4.13" evaluate scikit-learn

# 1) Models
huggingface-cli download Mati83moni/PLLuM-12B-chat-2512-EXL3 --local-dir exl3
huggingface-cli download CYFRAGOVPL/PLLuM-12B-chat-2512 --local-dir bf16 --include "*.safetensors" "*.json" "tokenizer*" "*.jinja"

# 2) Evaluation texts (CPU): Polish Wikipedia (20231101.pl, revision b04c8d1ceb2f, first 60 articles >= 3,000 chars)
#    and WikiText-2 test; SHA-256 is verified. Then the overlap check against the calibration corpus.
python benchmarks/make_eval_texts.py --out-dir eval_texts --meta
python benchmarks/calibration_overlap.py --exllamav3-repo exllamav3 --texts eval_texts/pl_wiki_eval.txt eval_texts/en_wikitext2_test.txt

# 3) Kernel check (fused EXL3 GEMM vs reconstruct path, all linear layers)
python benchmarks/validate_kernels.py --model exl3 --out validate.json

# 4) PPL / KLD / top-K vs original, ctx 2048, 100 rows (upstream eval/model_diff.py, text source swapped), weight SQNR
python benchmarks/model_diff_text.py --exllamav3-repo exllamav3 --model-a exl3 --model-b bf16 --rows 100 --text eval_texts/pl_wiki_eval.txt
python benchmarks/model_diff_text.py --exllamav3-repo exllamav3 --model-a exl3 --model-b bf16 --rows 100 --text eval_texts/en_wikitext2_test.txt
python benchmarks/weight_sqnr.py --exl3 exl3 --bf16 bf16 --out weight_sqnr.json

# 5) Polish tasks, 0-shot: EXL3 on one GPU, original BF16 split over 2x T4
python benchmarks/run_lm_eval.py --model exl3 --out lm_eval_exl3
python benchmarks/run_lm_eval.py --model bf16 --split --out lm_eval_bf16

# 6) VRAM / speed, one process per context length
for c in 2048 4096 8192 16384 32768; do python benchmarks/perf_ctx.py --model exl3 --ctx $c --out perf; done

# 7) Qualitative prompts (benchmarks/prompts_pl.json)
python benchmarks/qualitative.py --model exl3 --out qualitative_pl.json
python benchmarks/qualitative.py --model bf16 --split --out qualitative_pl_bf16.json

---

⚠️ Limitations

• Calibration: default ExLlamaV3 calibration (250 rows × 2048 tokens; mostly English web/wiki/code with a small multilingual part, a few Polish paragraphs). No Polish-specific calibration was used.
• One GPU, patched kernels: all measurements are on a Tesla T4 with the sm_75 patch. Quality metrics do not depend on the GPU beyond FP16 rounding; speed numbers do and are not representative of sm_80+ GPUs.
• Original model as FP16: the "original" in all comparisons is the BF16 checkpoint executed with FP16 activations by ExLlamaV3 (layer by layer for PPL/KLD, split over 2× T4 for the tasks), not a BF16 Transformers run.
• Not measured: few-shot and chat-template variants of the tasks, contexts > 2048 for PPL/KLD, contexts > 32,768 for VRAM, batch size > 1, quantized KV cache, TabbyAPI end-to-end serving, other GPUs.
• Benchmark caveats: PolEmo2 is scored by regex on generated text (format failures count as wrong); INCLUDE-Polish has small STEM (48) and social-science (4) subsets; ARC-Challenge-PL is machine-translated; Wikipedia may be in the training data of the original model; neither evaluation text shares any 13-word sequence with the calibration set (0 PL / 0 EN).

---

🔧 Quantization Process

1. Download original BF16 model from CYFRAGOVPL/PLLuM-12B-chat-2512
2. Quantize using ExLlamaV3 convert.py with target BPW 4.5
3. Verify output quality: the ExLlamaV3 conversion tool reported an average SQNR of ~33 dB during quantization. This value comes from the converter and was not reproduced in the measurements on this card (the independently measured weight-space SQNR is in Quality Assessment; full results in Benchmarks)

# Quantization command
python convert.py \
    -i /path/to/PLLuM-12B-chat-2512 \
    -w /path/to/work_dir \
    -o /path/to/PLLuM-12B-chat-2512-EXL3-4.5bpw \
    -b 4.5

Environment

---

📝 Notes

• Quantized with the default calibration dataset (250 rows, 2048 columns)
• EXL3 is the native quantization format of ExLlamaV3; no speed comparison with other formats was measured here
• Polish quality: Δ PPL +1.15 % on Polish Wikipedia vs the original weights (see Benchmarks)
• Minimum required version: ExLlamaV3 v0.0.43+ or TabbyAPI with exllamav3 backend

---

📜 License

This quantization is distributed under the Apache 2.0 License, inherited from the original model.

---

🔗 Links

• Original model: CYFRAGOVPL/PLLuM-12B-chat-2512
• PLLuM project: pllum.org.pl
• PLLuM paper: arXiv:2511.03823
• ExLlamaV3: turboderp-org/exllamav3
• TabbyAPI: theroyallab/tabbyAPI
• PLLuM-70B GGUF (also by Mati83moni): Mati83moni/Llama-PLLuM-70B-instruct-2512-GGUF
• Quantized by: @Mati83moni

---

🙏 Credits

• CYFRA.GOV.PL / HIVE AI — original PLLuM model training & research
• turboderp & ExLlamaV3 contributors — EXL3 format and quantization tools
• theroyallab — TabbyAPI inference server
• Mati83moni — EXL3 quantization and community distribution