Nanbeige4.1-3B-W8A8-INT8

Nanbeige4.1-3B - W8A8-INT8

Nanbeige/Nanbeige4.1-3B quantized to W8A8-INT8 (8-bit weights).

What this is

8-bit for Ampere (A100/A40/A6000), which has no fast FP8 path.

Caveat. Needs calibration, unlike FP8.

Details

Usage

vllm serve Sohailhosseini/Nanbeige4.1-3B-W8A8-INT8 \
  --max-model-len 32768

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM("Sohailhosseini/Nanbeige4.1-3B-W8A8-INT8", max_model_len=32768)
    out = llm.chat(
        [{"role": "user", "content": "What is quantization? Answer in one sentence."}],
        SamplingParams(temperature=0.6, max_tokens=512),
    )
    print(out[0].outputs[0].text)

Measured quality

Served under vLLM 0.27.1 (Python 3.12, torch 2.13.0+cu130, CUDA 13, NVIDIA A40) and driven through the OpenAI-compatible /v1/chat/completions endpoint - not merely loaded.

Three prompts were also run against the unquantized source on the same stack with greedy decoding (temperature=0). The quantized model reproduced the source's output token for token on all three, including an idiosyncratic token the base model itself emits. Faithful to the original on this sample; it is a sample, not a benchmark.

Provenance

Produced with HF-quantized. recipe.yaml in this repo is the exact modifier stack that was applied, and the scheme, ignored layers, calibration set and hardware are in the table above.

Licence is inherited from the source model. Quantization does not change what you are permitted to do with the weights.