Downloads · 30 days
132
15% of all-time downloads
punalmanalan/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound
Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound is a image-text-to-text model from punalmanalan. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
This repository is a W4A16 quantized derivative of:
Downloads · 30 days
132
15% of all-time downloads
All-time downloads
852
Public
Parameters
6.3B
22.3 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors17.5 GB · 76%
How the weights are stored.
I3227.5B · 98%
From the Hugging Face model README
This repository is a W4A16 quantized derivative of:
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
Original model:
https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
This is the same underlying Qwen3.8-27B multimodal model in a lower-bit inference format.
This repository is not a new fine-tune, merge, or model branch.
The model weights were quantized from the original DavidAU checkpoint for more efficient inference on GPUs with limited VRAM.
The original model's multimodal capability is retained. The vision component was not converted into a text-only model.
The 64 transformer language layers were quantized to:
Calibration and optimization:
| Setting | Value |
|---|---|
| AutoRound version | 0.15.0 |
| Dataset | NeelNanda/pile-10k |
| Calibration samples | 128 |
| Optimization iterations | 200 |
| Sequence length | 2048 |
| Batch size | 1 |
| Gradient accumulation | 8 |
| Torch compile | enabled on GPU |
| Disk streaming | enabled |
| Quantization device map | GPU |
The language-model output head, embeddings, and MTP components were prepared separately from the main 4-bit transformer body.
The MTP linear components were quantized separately and preserved in the final model package.
The final package also includes the required packed draft LM head and:
mtp_draft_vocab_ids.pt
for MTP/speculative decoding setups that support this format.
Base model:
Qwen/Qwen3.8-27B
Original fine-tuned model:
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
This repository:
punalmanalan/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound
Relationship:
Qwen/Qwen3.8-27B
↓
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
↓
W4A16 AutoRound quantization
↓
This repository
The original model is an Image-Text-to-Text model.
This quantized repository preserves the multimodal model structure and is not intended to be treated as a text-only checkpoint.
The primary Hugging Face task is therefore:
image-text-to-text
Example high-level Transformers usage:
from transformers import pipeline
pipe = pipeline(
"image-text-to-text",
model="punalmanalan/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound"
)
For runtimes that support the underlying multimodal architecture, image and text inputs can be supplied through the processor/model interface.
Runtime support for vision input depends on the inference framework and its support for this quantized format.
The original model supports a native context length of 262,144 tokens.
This quantized package was prepared for long-context inference and can be used with serving configurations above the native window only when the selected runtime and RoPE/context configuration explicitly support it.
Do not assume that increasing max_model_len automatically makes every runtime configuration safe or performant.
This quantization is intended for efficient local inference, particularly on consumer GPUs where the original BF16 checkpoint is expensive to load.
Typical use cases include:
Quantization necessarily changes the numerical representation of the model weights, so this checkpoint is not mathematically identical to the original BF16 model.
This repository was prepared specifically to reduce that loss while providing substantially lower memory usage.
No claim is made here that the quantized checkpoint is benchmark-identical to BF16. Users should evaluate both versions on their own workload when exact quality comparisons are important.
This package is intended primarily for inference frameworks that support:
compressed-tensorsIt was prepared and tested around a patched vLLM 0.27.1 serving stack with MTP support.
The exact runtime configuration used for this repository is documented separately in the associated deployment project.
The repository contains:
model_extra_tensors.safetensorsmodel.safetensors.index.jsonconfig.jsonThis repository is a quantization of the original DavidAU model.
It is not a new fine-tune.
It is not a new merge.
It is not the BIGBANG model.
It is not the NOESIS model.
It is the original:
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
converted to a W4A16 AutoRound inference format.
Original model:
DavidAU
Original model page:
https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
Quantization and packaging:
Punal Manalan
Quantization tooling:
Intel AutoRound
Calibration dataset:
NeelNanda/pile-10k