InternVL3_5-38B-Flash-FP8

InternVL3_5-38B-Flash-FP8

This repository contains an FP8 quantized version of the multi-modal model OpenGVLab/InternVL3_5-38B-Flash.

Model Details

• Base Model: OpenGVLab/InternVL3_5-38B-Flash
• Quantization Format: FP8 (E4M3) targeting Linear layers
• Quantization Engine: llmcompressor (Post-Training Quantization / PTQ)
• Calibration Dataset: 512 samples from ultrachat-200k (train_sft)
• License: MIT

Method & Creation Process

The model was quantized using Neural Magic's llm-compressor framework.

Post-training quantization (PTQ) was applied directly to the language backbone (model.language_model). This strategy ensures that all heavy Linear projections in the main LLM layers are converted to FP8 for maximum speedup and reduced VRAM footprint, while preserving the vision architecture intact.

Quantization Recipe

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8",
    ignore=["lm_head"]
)

oneshot(
    model=model.language_model, 
    tokenizer=tokenizer,
    dataset="ultrachat-200k",
    splits="train_sft[:512]",
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
)