BTL-4-Q4_K_M-GGUF

gopi87/BTL-4-Q4_K_M-GGUF

This model was converted to GGUF format from badtheorylabs/BTL-4 using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

Fix: BTL-4 GGUF fails to load — tensor 'blk.40.attn_norm.weight' not found

Symptom

Loading btl-4-q8_0.gguf in llama.cpp (or ik_llama.cpp) fails with:

llama_model_load: error loading model: check_tensor_dims: tensor 'blk.40.attn_norm.weight' not found
llama_model_load_from_file_impl: failed to load model

This happens on every loader tried (mainline llama.cpp, ik_llama.cpp), including current builds that already support Qwen3.5/3.6-style NextN/MTP models.

Root cause

The model's general.architecture is qwen35moe, and its GGUF metadata declares:

qwen35moe.block_count          = 41
qwen35moe.nextn_predict_layers = 1

This tells the loader: "40 normal transformer layers (0–39), plus 1 extra NextN/MTP speculative-decoding head layer (block 40)."

The loader walks through blocks 0–39 fine (confirmed in verbose logs — all the expected attn_norm, attn_qkv/attn_q/attn_k/attn_v, ssm_*, ffn_*_exps tensors load correctly). It then reaches block 40, expects to find NextN-head tensors there, and fails immediately on the first lookup.

Verification with gguf_dump.py confirmed block 40 has zero tensors in the file:

python3 gguf-py/gguf/scripts/gguf_dump.py /mnt/nvme/btl-4-q8_0.gguf 2>&1 | grep -oP "blk\.\d+" | sort -t. -k2 -n -u | tail -5
# blk.35
# blk.36
# blk.37
# blk.38
# blk.39

So this is not a loader-compatibility problem. The GGUF's metadata claims a NextN/MTP head exists, but the file was converted/quantized without ever writing the weights for it. There's nothing to recover or rename — the data simply isn't there.

Fix

Since the NextN layer has no data anyway, correct the metadata to describe the model as it actually is: a plain 40-layer model, no MTP head. This only patches two u32 metadata fields in place — no re-quantization, no touching the 34 GB of tensor data.

1. Back up the file first

cp /mnt/nvme/btl-4-q8_0.gguf /mnt/nvme/btl-4-q8_0.gguf.bak

2. Locate gguf_set_metadata.py in your llama.cpp checkout

find ~/llama.cpp -iname "gguf_set_metadata.py"
# typically: ~/llama.cpp/gguf-py/gguf/scripts/gguf_set_metadata.py

Activate the repo's venv if it has one (source venv/bin/activate).

3. Dry-run the changes to confirm the tool sees the right fields

python3 gguf-py/gguf/scripts/gguf_set_metadata.py \
  /mnt/nvme/btl-4-q8_0.gguf qwen35moe.block_count 40 --dry-run --verbose

python3 gguf-py/gguf/scripts/gguf_set_metadata.py \
  /mnt/nvme/btl-4-q8_0.gguf qwen35moe.nextn_predict_layers 0 --dry-run --verbose

Expected output:

* Preparing to change field 'qwen35moe.block_count' from 41 to 40
* Preparing to change field 'qwen35moe.nextn_predict_layers' from 1 to 0

4. Apply for real

python3 gguf-py/gguf/scripts/gguf_set_metadata.py \
  /mnt/nvme/btl-4-q8_0.gguf qwen35moe.block_count 40 --force

python3 gguf-py/gguf/scripts/gguf_set_metadata.py \
  /mnt/nvme/btl-4-q8_0.gguf qwen35moe.nextn_predict_layers 0 --force

Each should report Field changed. Successful completion.

5. Load normally

CUDA_VISIBLE_DEVICES=2,3,0,1 \
numactl --interleave=all \
~/llama.cpp/build/bin/llama-server \
    --model /mnt/nvme/btl-4-q8_0.gguf \
    --tensor-split 1.2,1.8,0.45,0.35 \
    --n-cpu-moe 99 \
    --ctx-size 280000 \
    --batch-size 6000 \
    --ubatch-size 6000 \
    --parallel 1 \
    --threads 42 \
    --threads-batch 42 \
    -ngl 100 \
    --host 127.0.0.1 \
    --port 8082 \
    --jinja

print_info should now show n_layer = 40 / n_layer_all = 40 (matching), and the server should load through to HTTP server listening without touching block 40 at all.

Trade-off

This model loses the speculative-decoding speedup that a working NextN/MTP head would have given (roughly 15–70% faster decode depending on hardware, per community benchmarks on similar Qwen3.5/3.6-class models). Base inference quality is unaffected — layers 0–39 (the actual model weights) are untouched.

If you want MTP working properly

The NextN head weights would need to be re-generated from the original Ornith-1.0-35B base model checkpoint (in whatever training framework Bad Theory Labs used) and the GGUF re-converted with a NextN-aware convert_hf_to_gguf.py that actually writes block 40's tensors — this metadata patch does not add that capability, it only stops the loader from looking for data that was never written.

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo gopi87/BTL-4-Q4_K_M-GGUF --hf-file btl-4-q4_k_m.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo gopi87/BTL-4-Q4_K_M-GGUF --hf-file btl-4-q4_k_m.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo gopi87/BTL-4-Q4_K_M-GGUF --hf-file btl-4-q4_k_m.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo gopi87/BTL-4-Q4_K_M-GGUF --hf-file btl-4-q4_k_m.gguf -c 2048