Qwen3.8-27B-1Bit-YaRN-1M

Qwen3.8-27B — 1-bit Quantization + YaRN 1M-Context Pipeline

Reproducible pipeline that takes Qwen3.8-27B (native 256K context) from HF safetensors and produces 1-bit (IQ1) GGUFs with YaRN 1M-token context scaling, plus an abliterated/uncensored variant.

This repo is a reusable recipe: swap MODEL_DIR for any future Qwen3-style model and rerun.

---

Live on Ollama

All four variants are published and usable directly via ollama run:

All four carry the YaRN 4x bake (native 262144 → 1M context) in their GGUF metadata, and num_ctx 1048576 is set in their Ollama Modelfiles.

---

Artifacts produced

Sizes include the high-precision MTP head (blk.64.*, q8_0), embedding (q8_0) and output (q6_k) tensors, which account for ~5.7 GB of each file. The GGUF files are published through Ollama (above); this repo documents the pipeline.

---

Workflow overview

HF safetensors (Qwen3.8-27B, arch = qwen35)
        │
        │ 1. convert_hf.sh            convert_hf_to_gguf.py --outtype f16
        ▼
   F16 GGUF (qwen3.8-27b-f16.gguf)
        │
        │ 2. run_imatrix.sh           llama-imatrix (CUDA build), ~500MB+ corpus
        ▼
   imatrix GGUF
        │
        │ 3. quant_base.sh            llama-quantize → IQ1_M / IQ1_S / UD-IQ1_M
        ▼
   1-bit GGUFs, YaRN 1M baked in

Four stages:

1. Convert — convert_hf_to_gguf.py turns the HF safetensors into an F16 GGUF. Qwen3.8-27B uses architecture Qwen3_5ForConditionalGeneration, which llama.cpp maps to GGUF arch qwen35 (this is why the override keys are qwen35.*).
2. Importance matrix — llama-imatrix computes per-tensor importance from a text corpus. Needs a CUDA build of llama.cpp. The imatrix GGUF is fed to llama-quantize via --imatrix.
3. 1-bit quantization — llama-quantize with IQ1_M/IQ1_S and the imatrix.
4. YaRN bake — the --override-kv flags embed YaRN 1M scaling directly into the GGUF metadata at quantization time, so no runtime flags are needed.

---

Requirements

• llama.cpp with CUDA (for llama-imatrix; quantization itself is CPU-friendly but the imatrix stage wants a GPU):

  cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j
  

• Corpus for the imatrix — ~500MB+ of text recommended (llama.cpp uses --chunks N --ctx-size 4096 --batch-size 2048). This repo uses ~818MB (wikitext-103-raw + two Parquet dumps concatenated into corpus/corpus.txt).
• Disk: ~26 GB for the base F16, plus another ~26 GB for the ablit F16, plus ~20 GB of quants. Plan for ~75–80 GB total during a full run.
• VRAM: the imatrix step loads the F16 model into VRAM (-ngl 99), so a 24GB+ GPU is effectively required (26GB F16 + KV). CUDA must be available.
• Python 3 + torch/transformers present in the llama.cpp env for the converter.

---

Exact commands

0. Environment

export HF_TOKEN=$(cat ~/.cache/huggingface/token)   # or set your token

1. Convert HF → F16 GGUF

./convert_hf.sh /workspace/parshu/model /workspace/parshu/gguf/qwen3.8-27b-f16.gguf
./convert_hf.sh /workspace/parshu/model_ablit /workspace/parshu/gguf/qwen3.8-27b-ablit-f16.gguf

Equivalent to:

python3 /workspace/parshu/llama.cpp/convert_hf_to_gguf.py /workspace/parshu/model \
    --outtype f16 --outfile /workspace/parshu/gguf/qwen3.8-27b-f16.gguf

2. Generate the importance matrix

/workspace/parshu/llama.cpp/build/bin/llama-imatrix \
    -m /workspace/parshu/gguf/qwen3.8-27b-f16.gguf \
    -f /workspace/parshu/corpus/corpus.txt \
    -o /workspace/parshu/gguf/imatrix-qwen3.8-27b.gguf \
    --output-format gguf --chunks 512 --ctx-size 4096 --batch-size 2048 \
    -ngl 99 --no-ppl

3. Quantize

Base variants:

LL=/workspace/parshu/llama.cpp/build/bin/llama-quantize
IM=/workspace/parshu/gguf/imatrix-qwen3.8-27b.gguf
IN=/workspace/parshu/gguf/qwen3.8-27b-f16.gguf
Y="--override-kv qwen35.rope.scaling.type=str:yarn --override-kv qwen35.rope.scaling.factor=float:4.0 --override-kv qwen35.rope.scaling.original_context_length=int:262144 --override-kv qwen35.rope.scaling.finetuned=bool:true"

# IQ1_M
$LL --imatrix $IM $Y --token-embedding-type q8_0 --output-tensor-type q6_k \
  --tensor-type 'blk\.64\..*weight$=q8_0' \
  $IN /workspace/parshu/gguf/qwen3.8-27b-IQ1_M-yarn.gguf IQ1_M 16

# IQ1_S
$LL --imatrix $IM $Y --token-embedding-type q8_0 --output-tensor-type q6_k \
  --tensor-type 'blk\.64\..*weight$=q8_0' \
  $IN /workspace/parshu/gguf/qwen3.8-27b-IQ1_S-yarn.gguf IQ1_S 16

# UD-IQ1_M (dynamic: attn q/k/v = Q6_K, attn_output = Q4_K, MLP = IQ1_M)
$LL --imatrix $IM $Y --token-embedding-type q8_0 --output-tensor-type q6_k \
  --tensor-type 'blk\.64\..*weight$=q8_0' \
  --tensor-type '.*\.attn_q\.weight$=q6_k' \
  --tensor-type '.*\.attn_k\.weight$=q6_k' \
  --tensor-type '.*\.attn_v\.weight$=q6_k' \
  --tensor-type '.*\.attn_output\.weight$=q4_k' \
  $IN /workspace/parshu/gguf/qwen3.8-27b-UD-IQ1_M-yarn.gguf IQ1_M 16

Abliterated variant:

$LL --imatrix $IM $Y --token-embedding-type q8_0 --output-tensor-type q6_k \
  --tensor-type 'blk\.64\..*weight$=q8_0' \
  /workspace/parshu/gguf/qwen3.8-27b-ablit-f16.gguf \
  /workspace/parshu/gguf/qwen3.8-27b-ablit-IQ1_M-yarn.gguf IQ1_M 16

All four runs happen in parallel (see quant_base.sh / quant_ablit.sh).

---

Why the MTP blk.64.* tensor-type override is required

Qwen3.8-27B ships a Multi-Token-Prediction (MTP) head at block 64. The F16 GGUF includes its tensors, but the imatrix contains no entries for them (they are not exercised by llama-imatrix). Without a manual --tensor-type override, llama-quantize fails for the IQ1 codebooks because IQ1 requires imatrix statistics for every tensor.

Workaround (attribution: bartowski):

--tensor-type 'blk\.64\..*weight$=q8_0'

This pins the MTP head weights to q8_0 so the quantizer skips them and never touches the missing imatrix entries. IQ1 quantization then succeeds.

---

YaRN 1M context (the --override-kv keys)

Qwen3.8-27B is native 256K. Qwen's official long-context recipe applies 4x YaRN scaling to reach 1M tokens — the same setting used in Qwen's own 1M-context models. These four keys are baked into each quantized GGUF at quantization time:

Because the values are baked into the GGUF metadata, no --override-kv or runtime flags are needed at inference — llama.cpp reads the scaling directly from the file.

Verify after quantization:

/workspace/parshu/llama.cpp/build/bin/llama-quantize --help   # or inspect with gguf-tools

---

UD (dynamic) quantization — attribution: unsloth

Unsloth's "UD" approach keeps attention projection tensors at higher precision because they are much more sensitive to quantization error, while dropping MLP weights to 1-bit. This variant uses:

Net effect: ~6.8 GB, slightly larger than plain IQ1_M, with better attention fidelity.

---

Inference

Any recent llama.cpp build (llama-cli, llama-server, llama.cpp bindings). Context length is baked in; just set -c (e.g. -c 1000000) and optionally -fa/flash attention:

llama-cli -m qwen3.8-27b-IQ1_M-yarn.gguf -c 1000000 -fa -p "Long prompt…"

---

Files

• convert_hf.sh — HF safetensors → F16 GGUF
• run_imatrix.sh — imatrix generation (CUDA llama.cpp)
• quant_base.sh — IQ1_M, IQ1_S, UD-IQ1_M from base F16 (YaRN baked)
• quant_ablit.sh — IQ1_M from abliterated F16 (YaRN baked)
• auto_pipeline.sh — watches quant completion → hash → blob → ollama create → push
• README.md — this document
• MODELCARD.md — per-variant model cards + quality disclaimer
• CREDITS.md — attribution