Llama-3.1-8B-Instruct-W8A8-FP8-KV

Llama-3.1-8B-Instruct-W8A8-FP8-KV

W8A8 FP8 per-tensor weights + activations + FP8 KV cache calibrated checkpoint of meta-llama/Llama-3.1-8B-Instruct.

Quantization details

• Scheme: W8A8 FP8 (E4M3) per-tensor, static activation scales
• KV cache: FP8 (E4M3) per-tensor, static calibrated scales
• Format: compressed-tensors
• Tool: llmcompressor==0.10.0.2
• Calibration: 512 samples from HuggingFaceH4/ultrachat_200k, sequence length 2048
• Seed: 42 (deterministic — re-running yields identical scales)

Why this exists

nvidia/Llama-3.1-8B-Instruct-FP8 ships W8A8 FP8 with k_scale = v_scale = 1.0 placeholders, so vLLM emits warnings and operates with identity KV-cache quantization scales. This checkpoint replaces the placeholders with values calibrated against ultrachat-200k, producing FP8 KV cache that actually preserves the K/V dynamic range.

Observed per-layer calibrated scales (32 layers total):

• k_scale: mean 0.0408, range [0.0236, 0.0757]
• v_scale: mean 0.0070, range [0.0011, 0.0148]

Use with vLLM

from vllm import LLM
llm = LLM(
    model="JongYeop/Llama-3.1-8B-Instruct-W8A8-FP8-KV",
    kv_cache_dtype="fp8_e4m3",
)

License

Released under the Llama 3.1 Community License.