Downloads · 30 days
44
11% of all-time downloads
Kimi-VL-A3B-Thinking-2506-W8A16
88plug/Kimi-VL-A3B-Thinking-2506-W8A16
Kimi-VL-A3B-Thinking-2506-W8A16 is a image-text-to-text model from 88plug. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
These weights are compressed-tensors (pack-quantized / int-quantized).
All-time downloads
397
Public
Parameters
16.5B
17.8 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors17.8 GB · 100%
Parameter types
How the weights are stored.
I815.3B · 92%
Try a prompt
Base models
Model card
From the Hugging Face model README
Load path (important)
These weights are compressed-tensors (pack-quantized / int-quantized).
| Runtime | Supported |
|---|---|
| vLLM ≥ 0.21 | Yes — preferred (auto-detect CT; no --quantization flag) |
transformers + compressed-tensors | Yes for many text models; multimodal may need custom code |
| Text Generation Inference (TGI) | Not supported for these CT packs |
| Hugging Face Inference Widget | Often fails — use vLLM locally instead |
# Preferred
vllm serve 88plug/<ModelName> --trust-remote-code
Do not deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors.
Kimi-VL-A3B-Thinking-2506-W8A16
INT8 post-training quantization of moonshotai/Kimi-VL-A3B-Thinking-2506 — a 16B MoE vision-language model with thinking/reasoning. Language-model Linear layers quantized W8A16; vision tower and merger kept BF16.
At a Glance
| Property | Value |
|---|---|
| Base model | moonshotai/Kimi-VL-A3B-Thinking-2506 |
| Release tier | Provisional (datafree RTN — re-quant scheduled) |
| Quant method | datafree RTN W8A16 (weight-only INT8) |
| FLAC status | Not measured (T+7d milestone) |
| Architecture | Vision encoder + merger + MoE language model |
| Quant format | compressed-tensors (native vLLM) |
| Quantized | language_model Linear layers |
| Kept BF16 | vision encoder, merger, projector, norms, router |
| Disk size | ~17 GB |
| Min GPU | 1× RTX A6000 48GB (recommended) |
Quick Start
Tested with vLLM v0.21.0. Weights are in compressed-tensors format — vLLM detects quantization automatically.
docker run --gpus device=0 -p 8080:8080 \
vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \
88plug/Kimi-VL-A3B-Thinking-2506-W8A16 \
--trust-remote-code \
--max-model-len 8192 \
--gpu-memory-utilization 0.90
Quantization Details
| Parameter | Value |
|---|---|
| Method | llmcompressor datafree |
| Weight bits | 8 (symmetric, group_size 128) |
| Activation | 16-bit |
actorder | N/A (datafree) |
| Recipe | recipe.yaml in repo |
Benchmarks
Benchmarks in progress.
License
MIT (per base model).
About
88plug AI Lab ships compressed-tensors quantizations for native vLLM v0.21.0+ deployment.
This release: Provisional tier — datafree RTN (weight-only rounding, no calibration corpus). A gold AutoRound re-quant is scheduled; 88plug architecture forbids new provisional W4A16 uploads.
Browse all releases → huggingface.co/88plug
- Task
- Image-Text-to-Text
- Library
- transformers
- Type
- kimi_vl
- License
- mit
- Languages
- en, zh
- Created
- Jul 1, 2026
- Updated
- Jul 13, 2026