Holo-3.1-35B-A3B-no-MTP-GGUF

Holo-3.1-35B-A3B GGUF Q4_K_M (no MTP)

This repository contains a GGUF conversion of Hcompany/Holo-3-1-35B-A3B for llama.cpp.

Files

• holo-3.1-35b-a3b-no-mtp-Q4_K_M.gguf — Q4_K_M quantized GGUF, ~20 GiB

Important note: no MTP

The upstream model config advertises:

"text_config": {
  "num_hidden_layers": 40,
  "mtp_num_hidden_layers": 1
}

However, the published safetensors/index only contain the base layers model.layers.0 through model.layers.39. No model.layers.40 / MTP tensors were found in the downloaded checkpoint.

If converted as-is, llama.cpp expects an extra MTP block and fails to load the GGUF with an error similar to:

missing tensor 'blk.40.attn_norm.weight'

For this conversion, text_config.mtp_num_hidden_layers was set to 0 before conversion. The resulting GGUF is a normal 40-block text model without the built-in MTP head.

Verified GGUF metadata:

general.architecture: qwen35moe
qwen35moe.block_count: 40
qwen35moe.nextn_predict_layers: absent
tensor_count: 733
max block: 39
has blk.40: false

Conversion details

• Source model: Hcompany/Holo-3-1-35B-A3B
• llama.cpp commit used locally: 5254a79
• Conversion: convert_hf_to_gguf.py --outtype f16
• Quantization: llama-quantize ... Q4_K_M
• F16 intermediate size: 69,376,637,088 bytes
• Q4_K_M output size: 21,166,757,728 bytes

Quantization summary:

model size = 66152.24 MiB / 16.01 BPW
quant size = 20175.71 MiB / 4.88 BPW

Example llama.cpp usage

llama-cli \
  -m holo-3.1-35b-a3b-no-mtp-Q4_K_M.gguf \
  -p "Hello" \
  -n 128 \
  -c 4096

For CUDA/P40-style serving, tune for your local build and VRAM budget, for example:

llama-server \
  -m holo-3.1-35b-a3b-no-mtp-Q4_K_M.gguf \
  -c 32768 \
  -ngl 999 \
  -ctk q5_0 \
  -ctv q5_0 \
  --flash-attn \
  --host 0.0.0.0 \
  --port 1235

Disclaimer

This is an unofficial community conversion/quantization. Please refer to the upstream model repository for license, intended use, and model details.