kappa-20b-131k-GGUF-Q8_0

kappa_20b_131k (GGUF Q8_0)

Q8_0 quantized GGUF of kappa_20b_131k for use with llama.cpp and compatible inference engines.

Part of the persona series — a set of experimental fine-tunes exploring personality-conditioned generation on a 20.9B MoE base.

This one (kappa) is full-parameter SFT at 131K context on multi-turn conversations with tool calling and 9 distinct personas. Built on OpenAI's GPT-OSS 20B base model. Trained on 4 desktop GPUs with torchtitan.

See also: BF16 GGUF (unquantized)

Files

Quantization Notes

Mixed-precision quantization: expert MLP weights are Q8_0 (8-bit integer), while attention weights (Q, K, V, O projections) are kept at BF16 to preserve attention quality. Biases, layernorms, router weights, and attention sinks remain in f32.

Q8_0 was chosen over k-quant variants (Q6_K, Q4_K_M) because the 3D expert weight tensors [2880, 2880, 32] don't meet k-quant block size requirements — 145 of 170 weight tensors fall back to higher precision, making Q6_K the same size as Q8_0 with no benefit.

Quantized from the BF16 source weights (not requantized from a prior quantization).

Model Details

Training

Full-parameter supervised fine-tuning (SFT) in bf16 — all 20.9B weights trainable, including every expert.

Hardware

4x NVIDIA RTX PRO 6000 Blackwell GPUs (96 GiB each) on a single workstation. Tensor parallelism degree 4. Peak memory utilization: 92.7 GiB per GPU (97.7%).

Training Framework

torchtitan with custom extensions for MoE, long-context packing, and CPU-offloaded optimization.

Persona System

The model was trained on multi-turn conversations across 9 robot personas mapped to the D&D alignment grid:

To activate a persona, set the system message to Persona: <alignment> (e.g., Persona: chaotic_evil). The model also works without a persona system message for general-purpose use.

Each persona maintains distinct behavioral characteristics while preserving task quality — the personality is in the delivery, not the substance.

Evaluation

RULER Long-Context Benchmark (131K)

Persona Alignment Grid

All 9 personas tested on identical prompts. Every persona provided complete, correct, and actionable responses while maintaining distinct character voice. Task quality was consistent across all alignments including the "evil" axis — no refusals or degraded helpfulness from any persona.

Sycophancy Resistance

Tested with 5 indirect sycophancy traps (false validation seeking, appeal to effort, false premises, social pressure after disagreement, false novelty claims). Results vary by persona:

• No persona: 3/5 resisted (caved on social pressure and effort-based flattery)
• lawful_evil: 5/5 resisted
• neutral_good: 4/5 resisted (mild softness on effort-based prompt)

Refusal Calibration

Tested with 10 prompts spanning legitimate edge cases and genuinely harmful requests:

• Correctly answered 8/8 legitimate requests (security research, medical information, historical analysis, fiction writing, lock picking, controversial opinions, dark humor)
• Correctly refused 2/2 harmful requests (phishing, drug synthesis)
• 1 borderline over-refusal (kitchen chemistry — refused the framing but still provided the explanation)

Usage

With llama.cpp

# Interactive chat (GPU offload)
llama-cli -m persona_kappa_20b_Q8_0.gguf-00001-of-00005.gguf -ngl 999

# Server mode
llama-server -m persona_kappa_20b_Q8_0.gguf-00001-of-00005.gguf -ngl 999 --port 8080

# With persona
llama-cli -m persona_kappa_20b_Q8_0.gguf-00001-of-00005.gguf -ngl 999 \
  --chat-template-file chat_template.jinja \
  -p "Persona: lawful_evil"

Known Quirks

• Persona training data is synthetic — some personas are stronger than others (chaotic_good tends to overcook catchphrases, neutral_evil voice can be weak)
• Can exhibit sycophancy under social pressure when used without a persona
• Over-refuses on some chemistry and safety-adjacent topics