Qwen3.8-27B-MTP-GGUF

Qwen3.8-27B MTP GGUF — Q4_K_M

Community GGUF Q4_K_M quantization of Qwen/Qwen3.8-27B with the original MTP / NextN tensors preserved.

This repository contains a format conversion and quantization of the original Qwen3.8-27B checkpoint.

No fine-tuning or additional training has been performed.

Model Details

Available File

Qwen3.8-27B-Q4_K_M-MTP.gguf

MTP / NextN

The original Qwen3.8-27B checkpoint contains Multi-Token Prediction / NextN tensors.

During conversion:

MTP enabled: 866 tensors
NextN disabled: 851 tensors
Difference: 15 MTP tensors

The 15 additional mtp.* tensors were intentionally preserved in this GGUF.

The current llama.cpp converter recognized the MTP export path and correctly mapped the NextN tensors into the additional model block.

llama.cpp Validation

The resulting Q4_K_M GGUF was successfully loaded and executed with llama.cpp using:

llama-cli \
  -m Qwen3.8-27B-Q4_K_M-MTP.gguf \
  --spec-type draft-mtp

The runtime successfully:

• loaded the GGUF;
• recognized the model architecture;
• recognized the embedded MTP / NextN tensors;
• constructed the main and MTP graphs;
• processed the prompt;
• generated output through the MTP-compatible runtime path.

Ollama

The GGUF has also been successfully imported and executed with Ollama 0.32.9.

Minimal Modelfile:

FROM ./Qwen3.8-27B-Q4_K_M-MTP.gguf

Create:

ollama create qwen3.8:27b-mtp-q4_K_M -f Modelfile

Run:

ollama run qwen3.8:27b-mtp-q4_K_M

MTP Configuration

For runtimes that expose MTP speculative decoding, the local TERATHOX configuration uses:

draft_num_predict = 4

Note that loading a GGUF containing MTP tensors does not by itself guarantee that a runtime is actively using speculative MTP decoding.

Users should verify MTP support and configuration for their specific runtime version.

Local TERATHOX Deployment

This quantization has been tested locally under the alias:

Terathox-Coder:Nova

Hardware

NVIDIA GeForce RTX 5080  16 GB
NVIDIA GeForce RTX 4070  12 GB
NVIDIA GeForce RTX 4070  12 GB

Three GPUs were used for the local validation.

Ollama Runtime Configuration

Context:                  204800
OLLAMA_FLASH_ATTENTION:   1
OLLAMA_VULKAN:            false
OLLAMA_KV_CACHE_TYPE:     q4_0
OLLAMA_SCHED_SPREAD:      false
OLLAMA_GPU_OVERHEAD:      0
OLLAMA_NUM_PARALLEL:      1
OLLAMA_MAX_LOADED_MODELS: 1
OLLAMA_KEEP_ALIVE:        -1
draft_num_predict:        4

Observed status:

NAME                   SIZE     PROCESSOR    CONTEXT
Terathox-Coder:Nova    24 GB    100% GPU     204800

Local Performance

Observed interactive generation performance:

Typical observed generation range:

~49–54 tokens/s

Prompt evaluation varied depending on conversation state and cached context, reaching values from approximately:

51 tok/s → 263 tok/s

These are local hardware measurements and not standardized model benchmarks.

Performance depends on hardware, context size, GPU offload, KV-cache configuration, runtime version and MTP implementation.

Intended Use

This GGUF is intended for:

• local text generation;
• coding and software engineering;
• agentic coding workflows;
• technical reasoning;
• long-context workloads;
• experimentation with MTP / NextN speculative decoding;
• local inference with llama.cpp or compatible GGUF runtimes.

Limitations

This is a quantized derivative of the original model.

Q4_K_M significantly reduces memory requirements but may introduce some quality degradation compared with the original BF16 checkpoint.

The base model may also produce inaccurate, biased or hallucinated information. Outputs should be independently verified for high-impact or safety-critical use cases.

Vision / Multimodal Support

The original Qwen3.8-27B model includes multimodal capabilities.

This repository currently provides the GGUF language-model artifact only.

No independently validated multimodal projector (mmproj) is currently included in this repository.

Therefore this release should currently be considered text-oriented unless an appropriate multimodal projector is added and validated.

Training

No training or fine-tuning was performed for this repository.

The original weights come from:

Qwen/Qwen3.8-27B

This repository only performs:

Original checkpoint
        ↓
GGUF BF16 with MTP preserved
        ↓
Q4_K_M quantization
        ↓
Qwen3.8-27B-Q4_K_M-MTP.gguf

Datasets

No additional dataset was used.

This repository does not contain a fine-tuned model.

Evaluation

No standardized quality benchmark was performed specifically on this quantization at the time of publication.

The performance results above measure local inference throughput only and should not be interpreted as accuracy or capability benchmarks.

For official capability benchmarks, refer to the original Qwen/Qwen3.8-27B model card.

Attribution

Original foundation model developed by the Qwen Team.

Base model:

Qwen/Qwen3.8-27B

GGUF conversion and Q4_K_M quantization:

Terathox-Coder

The original MTP / NextN tensors were preserved during conversion.

TERATHOX does not claim authorship or training of the original Qwen foundation model.

License

This repository follows the Apache License 2.0 of the base model.

Please review the original Qwen3.8-27B repository and license for additional information.

Disclaimer

This is a community conversion and is not an official Qwen release.

Compatibility, performance and MTP behavior may vary between versions of llama.cpp, Ollama and other GGUF runtimes.