deepseek-moe-16b-chat-FP8-GPTQ

DeepSeek-MoE-16B-Chat FP8 (GPTQ)

FP8 quantized version of deepseek-ai/deepseek-moe-16b-chat using GPTQ via llm-compressor.

Model Details

Quantization Recipe

Quantized with llm-compressor v0.11.0 on an NVIDIA H200 GPU.

from llmcompressor.modifiers.gptq import GPTQModifier

recipe = GPTQModifier(
    targets="Linear",
    scheme="FP8",
    ignore=["lm_head", "re:.*mlp\.gate$", "model.layers.0.mlp.down_proj"],
)

Ignored layers:

• lm_head — output projection, sensitive to quantization
• re:.*mlp\.gate$ — MoE router weights (64-expert routing decisions)
• model.layers.0.mlp.down_proj — dense first layer, column count (10944) not divisible by group size 128

Note: llm-compressor v0.11.0 has a known bug where actorder: "static" is written for channel/tensor-strategy quantization. vLLM v0.22.1+ rejects this. The config in this repo has been patched (actorder: null).

Serving with vLLM

pip install vllm
vllm serve soyrsoyr/deepseek-moe-16b-chat-FP8-GPTQ --max-model-len 2048 --trust-remote-code

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="soyrsoyr/deepseek-moe-16b-chat-FP8-GPTQ",
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}],
    max_tokens=128,
)
print(response.choices[0].message.content)

Collection

Part of DeepSeek-MoE-16B-Chat GPTQ Quantized — all four schemes (W4A16, W8A8, FP8, NVFP4) compared side by side.