gpt-oss-120b-Fable-5-Distilled-GGUF

How to Test GPT-OSS / Fable-5-Distilled GGUF Models Correctly

This document describes how to run HumanEval evaluation on GGUF-quantized GPT-OSS models (including Fable-5-Distilled) using llama-cpp-python. The key insight is that GPT-OSS uses a chat template with channels (analysis → final), which must be applied correctly for proper model behavior.

Test Results (HumanEval PASS@1)

The Q5_0 achieves a slightly higher score than Q8_0 on this benchmark (different quantization noise affects different problems). Fable-5-Distilled adds ~15-16% over the base model.

Prerequisites

pip install llama-cpp-python transformers datasets

A B300 GPU with ~275GB memory is sufficient for both models. The Q8_0 (115 GB) requires ~120 GB VRAM; the Q5_0 (67 GB) requires ~70 GB VRAM.

Full Evaluation Script

import json
import multiprocessing
import os
import re
import time

os.environ["CUDA_HOME"] = "/usr/local/cuda-13.0"
os.environ["PATH"] = f"/usr/local/cuda-13.0/bin:{os.environ.get('PATH', '')}"
os.environ["LD_LIBRARY_PATH"] = (
    "/home/user/.local/lib/python3.11/site-packages/nvidia/nccl/lib:"
    + os.environ.get("LD_LIBRARY_PATH", "")
)

from datasets import load_dataset
from llama_cpp import Llama
from vllm.tokenizers import get_tokenizer


def check_correctness(problem: dict, code: str, timeout: int = 15):

    def run_test(result_dict):
        try:
            exec_globals = {}
            exec(code, exec_globals)
            result_dict["passed"] = True
        except Exception:
            result_dict["passed"] = False

    manager = multiprocessing.Manager()
    result_dict = manager.dict()
    proc = multiprocessing.Process(target=run_test, args=(result_dict,))
    proc.start()
    proc.join(timeout)
    if proc.is_alive():
        proc.kill()
        manager.shutdown()
        return False
    passed = result_dict.get("passed", False)
    manager.shutdown()
    return passed


def extract_code_block(text: str) -> str | None:
    """Extract the first fenced Python code block (```python ... ```)."""
    blocks = re.findall(r"```python\s*\n(.*?)```", text, re.DOTALL)
    if not blocks:
        blocks = re.findall(r"```\s*\n(.*?)```", text, re.DOTALL)
    if blocks:
        for b in blocks:
            b = b.strip()
            if "def " in b or "import " in b or b.startswith("from "):
                return b
        return blocks[0].strip()
    return None


def generate_completion(llm: Llama, prompt: str) -> str:
    output = llm(
        prompt,
        max_tokens=8192,
        temperature=0.1,
        top_p=0.95,
        stop=[],
        echo=False,
    )
    return output["choices"][0]["text"]


def main():
    # 1. Load HumanEval
    print("Loading HumanEval dataset...")
    ds = load_dataset("openai_humaneval", split="test")
    print(f"Loaded {len(ds)} problems")

    # 2. Configure model paths
    gguf_path = "/path/to/model-Q8_0.gguf"
    tokenizer_path = "/path/to/original-model-dir"  # safetensors dir with tokenizer_config.json

    # 3. Load tokenizer (for chat template) and model
    print("Loading tokenizer...")
    tokenizer = get_tokenizer(tokenizer_path, trust_remote_code=True)

    print("Loading model...")
    start = time.time()
    llm = Llama(
        model_path=gguf_path,
        n_gpu_layers=-1,  # all layers on GPU
        n_ctx=4096,
        verbose=False,
        n_threads=16,
    )
    print(f"Model loaded in {time.time() - start:.1f}s")

    results = []
    passed = 0
    total = len(ds)

    for i, example in enumerate(ds):
        task_id = example["task_id"]
        prompt = example["prompt"]
        entry_point = example["entry_point"]
        test = example["test"]

        print(f"\n[{i + 1}/{total}] {task_id}")

        # 4. Apply GPT-OSS chat template (CRITICAL STEP)
        messages = [
            {
                "role": "system",
                "content": "You are a Python coding assistant. Complete the given function.",
            },
            {"role": "user", "content": prompt},
        ]
        formatted = tokenizer.apply_chat_template(
            messages, tokenize=False, add_generation_prompt=True
        )

        # The formatted prompt looks like:
        # <|start|>system<|message|>You are ChatGPT...
        # <|start|>developer<|message|># Instructions...<|end|>
        # <|start|>user<|message|>def foo(...):<|end|>
        # <|start|>assistant

        try:
            t0 = time.time()
            raw_output = generate_completion(llm, formatted)
            gen_time = time.time() - t0

            raw_output = raw_output.strip()

            # 5. Extract code from model output
            code = extract_code_block(raw_output)

            # Fallback: extract from final channel if no code fence found
            if not code:
                m = re.search(
                    r"<\|channel\|>final<\|message\|>(.*?)(?=<\|end\|>|<\|return\|>|$)",
                    raw_output,
                    re.DOTALL,
                )
                if m:
                    code = m.group(1).strip()
                else:
                    # Strip analysis channel and use remaining text
                    code = re.sub(
                        r"<\|start\|>assistant.*?<\|channel\|>final<\|message\|>",
                        "",
                        raw_output,
                        flags=re.DOTALL,
                    )
                    code = re.sub(
                        r"<\|channel\|>analysis<\|message\|>.*?<\|end\|>",
                        "",
                        code,
                        flags=re.DOTALL,
                    )
                    code = re.sub(
                        r"<\|channel\|>.*?<\|message\|>", "", code, flags=re.DOTALL
                    )
                    code = code.strip()

            if not code:
                code = raw_output.strip()

            # Clean up any remaining special tokens
            code = re.sub(r"^```(?:python)?\s*", "", code)
            code = re.sub(r"\s*```$", "", code)
            code = re.sub(r"<\|end\|>", "", code)
            code = re.sub(r"<\|return\|>", "", code)
            code = code.strip()

            # 6. Evaluate (use only the extracted code + test, no prompt duplication)
            t1 = time.time()
            correct = check_correctness(example, code + "\n" + test)
            check_time = time.time() - t1

            if correct:
                passed += 1
                print(f"  PASS ({gen_time:.1f}s gen, {check_time:.1f}s check)")
            else:
                print(f"  FAIL ({gen_time:.1f}s gen, {check_time:.1f}s check)")
                print(f"  Code snippet: {code[:100]}...")

            results.append(
                {
                    "task_id": task_id,
                    "prompt": prompt,
                    "raw_output": raw_output,
                    "completion": code,
                    "passed": correct,
                    "gen_time": gen_time,
                    "check_time": check_time,
                }
            )

        except Exception as e:
            print(f"  ERROR: {e}")
            results.append(
                {
                    "task_id": task_id,
                    "prompt": prompt,
                    "raw_output": "",
                    "completion": "",
                    "passed": False,
                    "gen_time": 0,
                    "check_time": 0,
                    "error": str(e),
                }
            )

        # Save intermediate results
        pass_rate = passed / (i + 1) * 100
        print(f"  Current: {passed}/{i + 1} = {pass_rate:.2f}%")

        with open("humaneval_results.jsonl", "w") as f:
            for r in results:
                f.write(json.dumps(r) + "\n")

    final_pass_rate = passed / total * 100
    print(f"\n{'=' * 40}")
    print(f"Final Results: {passed}/{total} = {final_pass_rate:.2f}%")
    print(f"{'=' * 40}")

    with open("humaneval_results.jsonl", "w") as f:
        for r in results:
            f.write(json.dumps(r) + "\n")


if __name__ == "__main__":
    main()

Key Steps Explained

1. Chat Template (CRITICAL)

The GPT-OSS model uses a specific Jinja chat template with channel-based output:

<|start|>system<|message|>...<|end|>
<|start|>developer<|message|># Instructions...<|end|>
<|start|>user<|message|>...<|end|>
<|start|>assistant

The model then generates:

<|channel|>analysis<|message|>...reasoning...<|end|>
<|start|>assistant<|channel|>final<|message|>...explanation...
```python
def foo():
    ...
```...<|end|>

Without this template, the model produces garbage (e.g., # TODO comments or natural language descriptions instead of code).

2. No Stop Tokens

Do not set stop=["<|end|>"] — the model emits <|end|> after the analysis channel. If you stop there, you miss the final channel with the actual code. Use stop=[] and rely on max_tokens or the model's native EOS token.

3. Code Extraction

• First look for ```python fenced blocks (the model often wraps code this way)
• If no fence, extract from <|channel|>final<|message|>
• Prefer code blocks that contain def or import (function definitions), not example usage blocks
• Always take the first code block matching these criteria, not the last (the model may include example usage in later blocks)

4. Evaluation Without Prompt

The extracted code typically includes the full function definition (the model regenerates the signature). Evaluate using only the extracted code + test:

correct = check_correctness(example, code + "\n" + test)

Do NOT prepend the prompt — doing so creates duplicate function definitions that cause exec errors.

5. Token Budget

Set max_tokens=8192. The model outputs analysis (reasoning) in the analysis channel before writing code. Complex problems may need 2000+ tokens for analysis alone, and another 500-2000 for the code.

Environment Notes

NCCL Compatibility

If you encounter undefined symbol: ncclCommWindowDeregister when importing transformers:

os.environ["LD_LIBRARY_PATH"] = (
    "/path/to/nvidia/nccl/lib:" + os.environ.get("LD_LIBRARY_PATH", "")
)

Use the NCCL bundled with PyTorch (at nvidia/nccl/lib/) rather than the system NCCL.

Loading the Tokenizer

Use vllm.tokenizers.get_tokenizer instead of transformers.AutoTokenizer to avoid triggering the torch CUDA import chain:

from vllm.tokenizers import get_tokenizer
tokenizer = get_tokenizer("/path/to/model", trust_remote_code=True)

Observed Results

Troubleshooting

Model outputs only analysis text, no code:

• Check that stop=[] (not stop=["<|end|>"])
• Increase max_tokens (8192+)
• The model may have naturally stopped — this is a model capability issue for certain problems

Model outputs # TODO comments instead of real code:

• The chat template is not being applied. The model needs the full GPT-OSS chat format to produce correct code.

Duplicate function definition errors:

• Don't prepend the prompt to the extracted code. The model generates the full function definition in its output.

GPT-OSS 120B Fable-5 Distilled — GGUF

Trained & Converted by: cloudyu
Compute Sponsor: AutoTrust AI Lab
Source Model: cloudyu/gpt-oss-120b-Fable-5-Distilled
Architecture: GPT-OSS (OpenAI MoE, 36 layers, 128 experts, 4 active)
Quantization: Q8_0 (115.7 GB) | Q5_0 (67 GB) — GGUF V3 for llama.cpp

GGUF is the universal format for LLM inference. These files run on llama.cpp, LM Studio, Ollama, GPT4All, text-generation-webui, llamafile, MLX-LM, and any GGUF-compatible runtime — across macOS, Windows, Linux, iOS, and Android, on CPU, CUDA, Metal, Vulkan, and ROCm backends.

Model Overview

This is a distilled variant of the OpenAI gpt-oss-120b model, fine-tuned using MLX with LoRA adapters (rank=16, targeting all attention projections, MoE router, and expert FFN layers). The training was performed in the MLX ecosystem using the MXFP4 quantized base model format. These GGUF files are the first community-produced GGUF conversions of a LoRA-fine-tuned GPT-OSS model from the MXFP4 format.

Available Quantizations

Weight matrices use Q8_0/Q5_0; biases, layer norms, and attention sinks are stored at full F32 precision for numerical stability.

Conversion Pipeline

The conversion from MLX MXFP4 format to llama.cpp GGUF involved a two-phase pipeline addressing several novel technical challenges:

Phase 1: MXFP4 Dequantization + LoRA Fusion → BF16

The source model is stored in Apple's MLX format using 4-bit MXFP4 quantization (Microscaling FP4, OCP spec) for the expert layers, with 8-bit affine quantization for attention projections. The LoRA adapter weights (576 tensors, float32) modify 288 modules spanning all Q/K/V/O projections, the MoE router, and all expert gate/up/down projections.

Key technical breakthroughs in Phase 1:

• Per-weight quantization dispatch: The model uses heterogeneous quantization parameters — MXFP4 4-bit (experts), affine 8-bit (attention + embeddings), and a special group_size=64 variant for the router. Each weight's dequantization must use the correct (bits, group_size) pair from the config.
• MXFP4 dequantization without explicit mode: MLX's dequantize function, when called with bits=4 and group_size=32, auto-detects the microscaling format. Explicit mode='mxfp4' is incompatible with the BF16 scale storage convention actually used.
• LoRA fusion with shape-dependent matmul: The MLX LoRA adapter stores lora_a as [input_dims, rank] and lora_b as [rank, output_dims] for 2D weights, but [num_experts, rank, input_dims] and [num_experts, output_dims, rank] for 3D expert weights. The fusion requires lora_b.T @ lora_a.T (2D) vs lora_b @ lora_a (3D) — using the wrong formula produces catastrophic dimension mismatches.
• BF16 overflow during MXFP4 dequant: The MXFP4 e8m0 scale representation can produce float32 values exceeding the BF16 representable range (~3.4e38), resulting in NaN values in ~5% of dequantized expert weights. These must be sanitized before downstream quantization.

Phase 2: BF16 → Quantized GGUF

Phase 2 reads the intermediate BF16 safetensors and writes the final GGUF in a single streaming pass, keeping RAM below 2 GB for a 218 GB intermediate.

Key technical breakthroughs in Phase 2:

• Missing attention.key_length metadata: The llama.cpp GPT-OSS handler computes n_embd_head_k = n_embd / n_head = 2880 / 64 = 45 by default, which is incorrect (the true head dimension is 64). This causes a fatal GGML_ASSERT(a->ne[0] == b->ne[0]) crash during KV cache initialization. The fix is explicitly setting gpt-oss.attention.key_length, value_length, key_length_swa, and value_length_swa to head_dim=64 in the GGUF metadata.
• Tokenizer vocabulary padding: The model's embedding table has 201,088 rows (vocab_size in config) but the o200k harmony tokenizer has only 200,019 entries. llama.cpp validates token_embd.weight against the tokenizer count. The token list must be padded with dummy tokens to reach 201,088.
• Attention sinks must be F32: The ggml_soft_max_add_sinks and ggml_flash_attn_ext_add_sinks operations require the sinks tensor to be GGML_TYPE_F32. Storing it as BF16 causes a Metal backend crash during inference.
• All biases must be F32: ggml_add operations in the attention and FFN graph require type-matched operands. Since attention outputs are F32, bias tensors stored as BF16 trigger binary_op: unsupported types errors on CPU and Metal.
• GGUF tensor naming alignment: The GPT-OSS architecture in llama.cpp (PR #15091, LLM_ARCH_OPENAI_MOE) maps MLX tensor names via specific conventions: self_attn.o_proj → attn_output (not attn_out), mlp.router → ffn_gate_inp, mlp.experts.gate_proj → ffn_gate_exps, self_attn.sinks → attn_sinks.weight, etc. The architecture identifier is gpt-oss (hyphenated), not gptoss.
• Safetensors byte-level data extraction: Reading individual tensors from safetensor files requires accounting for the JSON header size (f.seek(8 + header_length + offset), not f.seek(8 + offset)). Missing this causes the first tensor in each shard to read partially corrupt data — a silent data integrity bug.

Usage with llama.cpp

# Download (example path)
wget https://huggingface.co/cloudyu/gpt-oss-120b-Fable-5-Distilled/resolve/main/gpt-oss-120b-Fable-5-Distilled-Q8_0.gguf

# Inference
llama-cli -m gpt-oss-120b-Fable-5-Distilled-Q8_0.gguf \
  -p "Explain quantum computing in simple terms" -n 512

# Server (recommended — uses sliding window attention efficiently)
llama-server -m gpt-oss-120b-Fable-5-Distilled-Q8_0.gguf \
  --jinja --reasoning-format auto --port 8080

The model uses GPT-OSS's native reasoning format. Set reasoning_effort via chat template kwargs for API usage. The chat template supports system, developer, user, and assistant roles with channel markers for analysis, commentary, and final output.

Local Agent Best Practice

This model demonstrates extremely strong reasoning capabilities — when given --jinja and --reasoning-format auto with reasoning_effort: "high", it can perform multi-step planning, complex code analysis, and structured problem decomposition at a level comparable to frontier models. However, it has a critical weakness: severe hallucination. The model will confidently fabricate facts, API signatures, file paths, URLs, and library versions.

Golden Rule: Always pair this model with the anysearch skill in llama-agent or opencode, which grounds responses against real web search results. Do not trust any factual claim from this model without verification.

./build/bin/llama-agent \
  -m /path/to/gpt-oss-120b-Fable-5-Distilled-Q5_0.gguf \
  --jinja \
  --reasoning-format auto \
  --temp 1.0 \
  --chat-template-kwargs '{"reasoning_effort": "high"}'

Key flags explained:

• --jinja — Enables Jinja template rendering for GPT-OSS's multi-channel chat format
• --reasoning-format auto — Uses the model's native reasoning block structure (analysis / commentary / final)
• --temp 1.0 — Maximum temperature; strong reasoning models benefit from full creativity in the analysis phase
• --chat-template-kwargs '{"reasoning_effort": "high"}' — Instructs the model to allocate more compute to reasoning chains

Known failure modes without search grounding:

• Confidently invents non-existent CLI flags and API parameters
• Hallucinates plausible but wrong library versions and package names
• Fabricates convincing-looking URLs and file paths
• Constructs internally consistent but factually incorrect arguments

---

本地 Agent 最佳实践

本模型拥有极强的推理能力——配合 --jinja、--reasoning-format auto 和 reasoning_effort: "high" 参数，可以进行多步规划、复杂代码分析和结构化问题拆解，水平可比肩前沿模型。但它有一个致命弱点：严重的幻觉问题。模型会自信地捏造事实、API 签名、文件路径、URL 和库版本号。

黄金法则：始终将此模型与 anysearch skill 配合使用，以真实网页搜索结果作为事实锚定。未经查证，不要信任该模型的任何事实性结论。

./build/bin/llama-agent \
  -m /path/to/gpt-oss-120b-Fable-5-Distilled-Q5_0.gguf \
  --jinja \
  --reasoning-format auto \
  --temp 1.0 \
  --chat-template-kwargs '{"reasoning_effort": "high"}'

关键参数说明：

• --jinja — 启用 Jinja 模板渲染，支持 GPT-OSS 多通道 chat 格式
• --reasoning-format auto — 使用模型原生的推理块结构（analysis / commentary / final 三通道）
• --temp 1.0 — 最高温度；强推理模型在分析阶段受益于充分的创造性
• --chat-template-kwargs '{"reasoning_effort": "high"}' — 指示模型为推理链分配更多计算资源

未配合搜索验证时的已知失败模式：

• 自信地编造不存在的 CLI 参数和 API 接口
• 幻觉出看似合理但实际错误的库版本号和包名
• 构造看似可信的虚假 URL 和文件路径
• 生成内部自洽但与事实不符的论证

---

GPT-OSS 120B Fable-5 Distilled — GGUF（中文说明）

模型概要

训练与转换: cloudyu
算力支持: AutoTrust AI Lab

基于 OpenAI gpt-oss-120b 的蒸馏微调版本。使用 MLX 框架对 MXFP4 量化基座模型训练 LoRA 适配器（rank=16，覆盖所有注意力投影、MoE 路由器及专家 FFN 层）。这是社区首次将 LoRA 微调后的 GPT-OSS 模型从 MLX MXFP4 格式转换为 llama.cpp GGUF 格式。

量化版本

GGUF 是 LLM 推理的通用格式——支持 llama.cpp、LM Studio、Ollama、GPT4All、text-generation-webui、llamafile 等主流运行时，兼容 macOS/Windows/Linux/iOS/Android 全平台，CPU/CUDA/Metal/Vulkan/ROCm 全后端。

转换技术突破

阶段一：MXFP4 反量化 + LoRA 融合 → BF16

源模型以 MLX 格式存储，专家层使用 MXFP4（4-bit 显微缩放格式），注意力层使用 8-bit affine 量化。LoRA 适配器包含 576 个张量，修改 288 个模块。

核心技术挑战：

1. 异构量化参数调度：不同层使用不同的量化参数——专家层 bits=4, group_size=32 (mxfp4)，注意力层 bits=8, group_size=32 (affine)，路由器 bits=8, group_size=64。反量化必须按张量逐一使用正确的 (bits, group_size) 组合。

2. MXFP4 反量化的 mode 陷阱：mlx.dequantize 在 bits=4 时自动检测 MXFP4 格式，显式传 mode='mxfp4' 反而报错——因为它期望 uint8 格式的 scale，而实际存储的是 BF16。

3. LoRA 融合的形状相关矩阵乘法：2D 权重（注意力、路由器）与 3D 权重（专家）的 LoRA 融合公式不同——2D 用 lora_b.T @ lora_a.T，3D 用 lora_b @ lora_a。混用会导致维度崩溃。

4. BF16 溢出：MXFP4 的 e8m0 scale 编码可能产生超出 BF16 范围（±3.4e38）的 float32 值，导致约 5% 的专家权重变成 NaN。必须在量化前用 nan_to_num 清理。

阶段二：BF16 → 量化 GGUF

以流式方式读取 BF16 中间文件，逐张量量化写入 GGUF，内存占用 < 2 GB。

核心技术挑战：

1. attention.key_length 元数据缺失：llama.cpp 的 GPT-OSS 处理程序默认计算 n_embd_head_k = hidden_size / num_heads = 2880 / 64 = 45，但实际 head_dim 是 64。导致 KV cache 初始化时 ggml_set_rows assertion 崩溃（缓存 ne[0]=360 ≠ K张量 ne[0]=512）。修复方法是在 GGUF 元数据中显式设置 key_length/value_length/key_length_swa/value_length_swa = 64。

2. 词表补齐：模型 embedding 有 201,088 行，但 o200k harmony tokenizer 只有 200,019 个 token。llama.cpp 会校验 token_embd.weight 的形状与 tokenizer 数量必须一致。需用虚拟 token 补齐。

3. Attention Sinks 必须 F32：ggml_soft_max_add_sinks 操作要求 sinks 张量为 F32 类型，存为 BF16 会导致 Metal 后端推理崩溃。

4. 所有 biases 必须 F32：注意力输出是 F32，ggml_add 要求操作数类型匹配。BF16 存储的 bias 会在 CPU/Metal 后端触发 binary_op: unsupported types 错误。

5. GGUF 张量命名对齐：llama.cpp GPT-OSS 架构（PR #15091）的张量命名约定：attn_output（非 attn_out）、ffn_gate_inp、ffn_gate_exps、attn_sinks.weight 等。架构标识符为 gpt-oss（带连字符）。

6. Safetensors 字节级偏移：读取单个张量时必须计算 JSON header 的实际长度：f.seek(8 + header_length + offset)，而非简单的 f.seek(8 + offset)。每个 shard 的第一个张量会因忽略 header 长度而读到前一个 shard 的尾部垃圾数据。

使用方式

llama-cli -m gpt-oss-120b-Fable-5-Distilled-Q8_0.gguf -p "你好" -n 512

# API 服务（推荐）
llama-server -m gpt-oss-120b-Fable-5-Distilled-Q8_0.gguf \
  --jinja --reasoning-format auto --port 8080

模型使用 GPT-OSS 原生的 reasoning format，支持 reasoning_effort 参数控制推理深度，chat template 包含 analysis、commentary、final 三个输出通道。