WeMM-Embedding-9B-FP8

WeMM-Embedding-9B (FP8)

FP8 (E4M3) quantization of tencent/WeMM-Embedding-9B, produced with the same per-tensor round-to-nearest procedure used for the 2B variant.

• Method: per-tensor FP8 (E4M3, scale = absmax/448), applied to all nn.Linear layers — including the vision tower at 8-bit (safe). 4-bit vision-tower quantization is known to degrade multimodal retrieval, so it is intentionally avoided.
• Size: 18.8 GB (BF16) → 9.8 GB (FP8), ~2× smaller, near-lossless on retrieval benchmarks.
• Format: model.fp8.safetensors + fp8_scales.json (dequant at load: w = w_fp8 * scale). Loads with transformers / sentence-transformers using the same API as the BF16 base.

Why FP8 (not 4-bit)

The 9B model's multimodal quality is sensitive to vision-tower precision. FP8 (8-bit) keeps the vision tower well above the degradation threshold observed with 4-bit, while still halving weight memory — ideal for serving on FP8-capable GPUs (Ada / Hopper, e.g. RTX 4090, H100).

Usage (transformers / sentence-transformers)

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Weidows/WeMM-Embedding-9B-FP8", trust_remote_code=True)
emb = model.encode("Represent the meaning of this sentence.")

Usage (vLLM / SGLang)

FP8 is natively supported on Ada/Hopper GPUs. Serve with vLLM:

vllm serve Weidows/WeMM-Embedding-9B-FP8 --task embed

Evaluation

Quantized and evaluated with the identical per-tensor FP8 (E4M3) pipeline used for the 2B variant, on the same engine (transformers) as the BF16 baseline to isolate pure quantization loss.

COCO split: val2017, 200 images × 5 captions = 1001 captions (hard pool — random baseline ≈0.5% R@1). Every metric moves <0.002 absolute; FP8 rounding is benign at the 9B scale. Weight size: 18.8 GB → 9.8 GB (~2× compression), vision tower kept at 8-bit to avoid the 4-bit degradation path.