EVIE-Preview-4.5B-int8

EVIE-Preview-4.5B · bitsandbytes INT8

An unofficial 8-bit quantization of Tencent EVIE-Preview-4.5B, converted by Fanny-Eater.

This is a visual document retriever, not a chat or text-generation model. It returns one normalized 128-dimensional vector per token and ranks pages with MaxSim late interaction. It cannot be loaded by llama.cpp, LM Studio, Ollama, or any GGUF runtime — see Why not GGUF?.

All linear layers are quantized to INT8 via bitsandbytes LLM.int8() (load_in_8bit=True). There is no calibration dataset, no fine-tuning, and no retraining — this is a pure post-training weight-space quantization of the official checkpoint. All 725 weight tensors are preserved, including the custom_text_proj retrieval projection head (a naive AutoModel load silently drops it — this checkpoint does not).

Why not GGUF / llama.cpp / LM Studio?

llama.cpp's convert_hf_to_gguf.py does not support the ColQwen3_5 architecture (it errors with Model ColQwen3_5 is not supported). EVIE is a multi-vector retrieval encoder built on a Qwen3.5 hybrid (linear GatedDeltaNet + full attention) backbone with a vision tower — not a generative decoder — so GGUF tooling has no target for it. If you want an OpenAI-style endpoint instead of a local GUI, see Serving & benchmarking.

Quick start

pip install "git+https://github.com/huggingface/transformers.git" colpali-engine bitsandbytes accelerate

import torch
from colpali_engine.models import ColQwen3_5, ColQwen3_5Processor

model_id = "Fanny-Eater/EVIE-Preview-4.5B-int8"
model = ColQwen3_5.from_pretrained(model_id, device_map="cuda").eval()
processor = ColQwen3_5Processor.from_pretrained(model_id)

# Reproduce official scores: full-attention layers must be bidirectional
# (copy bidirectional.py from the source repo and call enable_bidirectional_attention(model))

from PIL import Image
images = [Image.open("page_1.png"), Image.open("page_2.png")]
queries = ["What is the revenue for 2025?"]

with torch.inference_mode():
    doc = model(**processor.process_images(images).to(model.device))
    model.rope_deltas = None
    qry = model(**processor.process_queries(queries).to(model.device))
    scores = processor.score(qry, doc)
print(scores)

The quantized weights reload identically from disk (verified: a text query yields an embedding of shape (1, 15, 128) in bf16). On a single 20 GB GPU the model loads with ample headroom.

Serving & benchmarking

There is no llama.cpp endpoint for this architecture. The practical ways to run and benchmark it yourself:

1. Python / colpali-engine (recommended) — the code above. For a real benchmark, run the official reproduce.sh from the source repo against this quantized checkpoint, or use the ViDoRe benchmark to compare against ColPali, ColQwen, Jina, and the other leaderboard models. This gives nDCG@5/@10, the metric the leaderboard uses.
2. FastAPI / embedding server — wrap the snippet above in a small FastAPI app to expose a POST /embed endpoint returning token vectors; point your retrieval pipeline at it. (Happy-path example in the discussion tab if there's demand.)
3. Baseline comparison — run the same queries against the official BF16 model and this INT8 one and diff the scores; expect differences well under 1 nDCG point.

Validation performed

• ✅ Strict round-trip: saved INT8 checkpoint reloads with all 725 tensors.
• ✅ Sanity forward pass on CPU: query embedding shape (1, 15, 128), finite, unit-normalized.
• ⚠️ No full ViDoRe leaderboard rerun was performed for this quantization. Treat benchmark parity as expected but unverified; LLM.int8() typically costs <0.1–0.5 nDCG on retrieval encoders. Run Serving & benchmarking step 1 to confirm on your hardware.

Notes & caveats

• bitsandbytes prints MatMul8bitLt: inputs will be cast from torch.bfloat16 to float16 on CPU — benign; on CUDA it runs the native int8 path.
• Quantization was done on CPU (device-agnostic); the saved checkpoint is hardware-independent.
• License: Apache-2.0, inherited from the source model. Credit Tencent for the model; this is a format conversion only.

Citation

@misc{evie-preview-4.5b,
  title  = {EVIE-Preview-4.5B},
  author = {Tencent},
  year   = {2026},
  url    = {https://huggingface.co/tencent/EVIE-Preview-4.5B}
}