OpenThinker3-7B-FP8

OpenThinker3-7B - FP8

open-thoughts/OpenThinker3-7B quantized to FP8 (8-bit weights).

What this is

Near-lossless, no calibration data, and it halves every Linear weight. The safe default when you care about quality and have Ada/Hopper or newer.

Caveat. Needs compute capability >= 8.9 (Ada/Hopper+) to run fast.

Details

Usage

vllm serve Sohailhosseini/OpenThinker3-7B-FP8 \
  --max-model-len 32768

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM("Sohailhosseini/OpenThinker3-7B-FP8", max_model_len=32768)
    out = llm.chat(
        [{"role": "user", "content": "What is quantization? Answer in one sentence."}],
        SamplingParams(temperature=0.6, max_tokens=512),
    )
    print(out[0].outputs[0].text)

Measured quality

Served under vLLM 0.27.1 (Python 3.12, torch 2.13.0+cu130, CUDA 13, NVIDIA RTX 4090) and driven through the OpenAI-compatible /v1/chat/completions endpoint - not merely loaded. Three prompts, greedy decoding, all three coherent.

Note the hardware: FP8 kernels need compute capability >= 8.9, so this was verified on Ada rather than the Ampere cards used to produce it. That is the same floor the caveat above describes.

Provenance

Produced with HF-quantized. recipe.yaml in this repo is the exact modifier stack that was applied, and the scheme, ignored layers and hardware are in the table above.

Licence is inherited from the source model. Quantization does not change what you are permitted to do with the weights.