Qwen3.6-27B-Text-NVFP4-MTP

Use Cases

Powerful and efficient local agentic coding.

NOTE: Text-only — ideal for a local LLM coding assistant.

With the llama-server settings below, it just fits in 32GB of VRAM with a 262,144 token context.

---

Compatibility

Tested environment:

---

Conversion Workflow

1. Checkout the required llama.cpp branch

cd .\llama.cpp
git fetch origin pull/21095/head:pr-21095
git checkout pr-21095
python -m pip install --no-cache-dir -r requirements\requirements-convert_hf_to_gguf.txt

2. Patch the tokenizer configuration

The upstream tokenizer configuration required a compatibility patch before conversion:

(Get-Content <model_dir>\tokenizer_config.json) `
  -replace '"tokenizer_class": "TokenizersBackend"', '"tokenizer_class": "Qwen2Tokenizer"' `
  | Set-Content <model_dir>\tokenizer_config.json

3. Convert the model

python convert_hf_to_gguf.py <model_dir> --outfile <output.gguf> --outtype bf16 --verbose

4. Transplant MTP blocks

As the MTP blocks were missing after conversion, used this gist to transplant MTP blocks from stevelikesrhino/Qwen3.6-27B-Q8-NVFP4-MTP-GGUF.

---

llama-server Settings

./llama-server ^
    --model ../models/llm/jdermody/Qwen3.6-27B-Text-NVFP4-MTP.gguf ^
    --alias "jdermody/Qwen3.6-27B-Text-NVFP4-MTP" ^
    --ctx-size 262144 ^
    --temp 0.6 ^
    --top-p 0.95 ^
    --min-p 0.00 ^
    --top-k 20 ^
    --port 8001 ^
    --presence-penalty 0.0 ^
    --repeat-penalty 1.0 ^
    --flash-attn on ^
    --jinja ^
    --batch-size 1024 ^
    --ubatch-size 512 ^
    --parallel 1 ^
    --cache-type-k q8_0 ^
    --cache-type-v q8_0 ^
    --spec-type draft-mtp ^
    --spec-draft-n-max 4 ^
    --spec-draft-type-k q4_0 ^
    --spec-draft-type-v q4_0