Downloads · 30 days
13
13% of all-time downloads
IKilledBardia/Qwen3.6-27B-FP8-unsloth
Qwen3.6-27B-FP8-unsloth is a machine learning model from IKilledBardia. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
FP8 quantization of Qwen/Qwen3.6-27B (a hybrid linear-attention + full-attention model), ready for vLLM.
Downloads · 30 days
13
13% of all-time downloads
All-time downloads
100
Public
Parameters
27.4B
58.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors30.4 GB · 100%
How the weights are stored.
F8_E4M324.3B · 89%
From the Hugging Face model README
FP8 quantization of Qwen/Qwen3.6-27B (a hybrid linear-attention + full-attention model), ready for vLLM.
float8_e4m3fn), block-wise [128, 128] scales (weight_scale_inv).quant_method: fp8, weight_block_size: [128,128], activation_scheme: dynamic.lm_head, embeddings, norms, and the SSM parameters (conv1d, in_proj_a/b, A_log, dt_bias) are kept in higher precision.--tasks mmlu, 5-shot, full test set, vLLM)| Model | MMLU acc |
|---|---|
| This model (block-wise FP8) | 0.8445 ± 0.0030 |
| Qwen/Qwen3.6-27B-FP8 (reference) | 0.8428 ± 0.0030 |
Per-task and raw results are in eval/.
from vllm import LLM
llm = LLM(model="IKilledBardia/Qwen3.6-27B-FP8-unsloth", trust_remote_code=True)
Requires a vLLM build that supports
Qwen3_5ForConditionalGenerationand is compiled against your CUDA runtime (e.g. vLLM 0.19.1 for CUDA 12.8).
An earlier revision of this repo used per-tensor modelopt FP8. Although its stored
weights were numerically faithful to the base model, vLLM's qwen3_5 execution path
mishandles the per-tensor FP8 format for this hybrid architecture, collapsing MMLU to
random (0.2325). Re-exporting in the block-wise [128,128] FP8 format above restores
full quality.