Xing4.0-29B-A4B-APEX-GGUF

Xing4.0-29B-A4B — Apex Quant GGUF Models

This repository contains 3 quantized GGUF profiles of the XingChen-AGI/Xing4.0-29B-A4B model, produced using Apex-Quant mixed-precision quantization technology.

Note: The original source model is available at XingChen-AGI/Xing4.0-29B-A4B under Apache 2.0 license. This repository provides quantized variants only.

📦 Quick Overview

BPW = Bits Per Weight. Higher value generally correlates with better generation quality.

📁 Repository Contents

├── Xing4.0-29B-A4B-i-quality.gguf   # 19.21 GB — highest quality
├── Xing4.0-29B-A4B-i-balanced.gguf  # 22 GB — balanced
├── Xing4.0-29B-A4B-i-compact.gguf   # 14 GB — compact
└── patches/
    └── 0001-xing4_0-llama-cpp-build.patch  # Reproducible build instructions

🏗️ Source Model Information

Key Architecture Features

• Multi-Latent Attention (MLA): DeepSeek-V3-style MLA compression with q_lora_rank=768, kv_lora_rank=512, enabling efficient attention without full projection matrices.
• Mixture-of-Experts (MoE): 64 routed experts per layer, top-4 active per token (noaux_tc gating method), plus 1 shared expert per layer.
• Masked Hyper-Connections (mHC): Sinkhorn-gated routing (mh_mult=4, sinkhorn_iters=20) for enhanced expert collaboration.
• Multi-Token Prediction (MTP): Draft head at layer 40 supporting speculative decoding.
• RoPE: YaRN positional embeddings with scaling factor 64, theta 10,000.

🔬 Quantization Methodology

All three profiles were generated using the Apex-Quant framework with importance-weighted mixed-precision quantization:

Apex-Quant Configuration

# Generate per-profile config (EDGE/NEAR/MID layer bands)
bash scripts/generate_config.sh \
  --profile i-quality \
  --layers 40 \
  --dense-layers 2 > config.txt

# Append MTP block patterns (blk.40) for draft-head quantization
echo "blk.40.ffn_gate_exps=iq4_xs" >> config.txt
echo "blk.40.ffn_up_exps=iq4_xs" >> config.txt
echo "blk.40.ffn_down_exps=iq4_xs" >> config.txt
# ... (see config files for full content)

# Quantize with imatrix
llama-quantize \
  --tensor-type-file config.txt \
  --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf \
  Xing4.0-29B-A4B-BF16.gguf \
  Xing4.0-29B-A4B-i-quality.gguf \
  Q6_K

Layer Band Allocation

Important Implementation Notes

1. MTP Block (blk.40): The multi-token prediction draft head has no imatrix entries because it is not executed during trunk-only inference. Types requiring imatrix (IQ1/IQ2 series) cannot be used for blk.40 tensors — this profile uses IQ4_XS/Q5_K/Q3_K which do not require imatrix.

2. mHC Gate Tensors: Automatically excluded from quantization by the llama.cpp fork (hc_attn_fn.weight, hc_ffn_fn.weight, hc_head_fn.weight preserved at F32).

3. MLA B-Matrices (attn_k_b): Tensors with shape [128, 512, 32] have an inner dimension (128) that is not divisible by Q6_K's block size (256). These automatically fall back to Q8_0 via the built-in fallback mechanism — a safe direction (higher precision).

4. Router Tensors (ffn_gate_inp): Fall through to the base type (Q6_K for i-quality, Q5_K for i-balanced, Q4_K for i-compact).

5. Config File Format: The --tensor-type-file parser does NOT support comment lines (#). Only pure pattern=value lines are valid.

💻 Inference Usage

Recommended Settings (from official repo)

[main]
ctx-size = 65536
flash-attn = true
cache-type-k = q8_0
cache-type-v = q8_0
n-gpu-layers = 999

llama.cpp CLI Examples

# i-compact — fastest, lowest VRAM requirement
.\llama-cli.exe ^
  -m models\Xing4.0-29B-A4B-i-compact.gguf ^
  -ngl 999 -c 65536 --flash-attn ^
  --temp 0.7 --top-k 50 --top-p 0.95 ^
  -p "Türkiye'nin en büyük şehri neresidir?" -n 64

# i-quality — highest accuracy
.\llama-cli.exe ^
  -m models\Xing4.0-29B-A4B-i-quality.gguf ^
  -ngl 999 -c 65536 --flash-attn ^
  --temp 0.7 --seed 42 ^
  -p "Explain quantum computing in simple terms." -n 64

# Server mode
.\llama-server.exe ^
  -m models\Xing4.0-29B-A4B-i-compact.gguf ^
  -ngl 999 -c 65536 --host 0.0.0.0 --port 8080

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="Xing4.0-29B-A4B-i-compact.gguf",
    n_ctx=65536,
    n_gpu_layers=999,
    flash_attn=True,
    seed=42,
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello, tell me about Istanbul."}],
    max_tokens=256,
)
print(response["choices"][0]["message"]["content"])

VRAM Sizing Guide

KV Cache: At 65,536 context with q8_0/kv ≈ 2.5 GB. To save VRAM, disable GPU KV offload with --no-kv-offload (moves KV cache to system RAM).

🛠️ Reproducible Build

To reproduce these quantizations from scratch:

Step 1: Clone and Build the Xing4.0 Fork

git clone -b xing4_0-port https://github.com/shuxiaoqiong/llama.cpp.git llama-cpp-xing4
cd llama-cpp-xing4

cmake -G "Visual Studio 17 2022" -A x64 ^
      -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON ^
      -B build
cd build
cmake --build . --config Release --parallel

Required binaries: llama-quantize.exe, llama-imatrix.exe, llama-cli.exe

Step 2: Convert Source Model to GGUF

python convert_hf_to_gguf.py ^
  <path-to-safetensors-dir> ^
  --outfile Xing4.0-29B-A4B-BF16.gguf ^
  --outtype bf16

Expected output: ~59 GB GGUF with 977 tensors.

Step 3: Generate Apex Configs

# Quality profile
bash scripts/generate_config.sh --profile i-quality --layers 40 --dense-layers 2 > configs/xing4_quality.txt

# Balanced profile
bash scripts/generate_config.sh --profile i-balanced --layers 40 --dense-layers 2 > configs/xing4_balanced.txt

# Compact profile
bash scripts/generate_config.sh --profile i-compact --layers 40 --dense-layers 2 > configs/xing4_compact.txt

Append MTP block (blk.40) patterns to each file:

blk.40.ffn_gate_exps=<EXP_TYPE>
blk.40.ffn_up_exps=<EXP_TYPE>
blk.40.ffn_down_exps=<EXP_TYPE>
blk.40.ffn_gate_shexp=<SHARED_TYPE>
blk.40.ffn_up_shexp=<SHARED_TYPE>
blk.40.ffn_down_shexp=<SHARED_TYPE>
blk.40.attn_q=<ATTN_TYPE>
blk.40.attn_k=<ATTN_TYPE>
blk.40.attn_v=<ATTN_TYPE>
blk.40.attn_output=<ATTN_TYPE>

Where <EXP_TYPE> = iq4_xs (quality) / Q5_K (balanced) / Q3_K (compact), and <ATTN_TYPE> = Q6_K (quality/balanced) / Q4_K (compact).

Step 4: Run Quantization

See the batch script at .tmp/apex_quant_xing4.ps1 for the automated 3-profile run, or invoke individually:

llama-quantize.exe ^
  --tensor-type-file configs\xing4_quality.txt ^
  --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf ^
  Xing4.0-29B-A4B-BF16.gguf ^
  Xing4.0-29B-A4B-i-quality.gguf ^
  Q6_K

📊 Technical Summary

Tensor Distribution Statistics

Quantization Impact

🙏 Acknowledgments

• localai-org/apex-quant — Apex-Quant MoE-aware mixed-precision quantization framework
• shuxiaoqiong/llama.cpp — Xing4.0 port fork (branch xing4_0-port) with native architecture support
• ggerganov/llama.cpp — GGUF format specification and quantization engine
• XingChen-AGI — Original Xing4.0-29B-A4B model creators (China Telecom / XingChen-AGI)
• jmarceno — Community imatrix and quantization recipe reference (imatrix-Xing4.0-29B-A4B-820chunks.gguf)

📝 Known Issues & Limitations

1. Smoke Test Output: Initial smoke testing with llama-cli returned whitespace-only output on CPU (-ngl 0) and partial GPU offload (-ngl 20). This may indicate MoE expert routing issues in certain runtime configurations. Further testing with different sampler parameters (--top-k, --min-p) and full GPU offload is recommended before production use.

2. MTP Draft Head: The draft head at blk.40 is loaded but not executed during standard inference. Speculative decoding support depends on the inference engine's MTP implementation.

3. Imatrix Dependency: While IQ4_XS, Q5_K, Q4_K, and Q3_K types work without imatrix, any future profiles using IQ1/IQ2 series will fail on blk.40 tensors due to missing imatrix entries for the MTP block.

📄 License

The original model XingChen-AGI/Xing4.0-29B-A4B is distributed under the Apache-2.0 license. These quantized derivatives are shared under the same license. See the original model card for full license terms and usage restrictions.

🔗 Related Links

• Original Model: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B
• Official GGUF Repo: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B-GGUF
• Deploy Tutorial: https://github.com/shuxiaoqiong/xing4_0-llama-pc-deploy
• Apex-Quant Framework: https://github.com/localai-org/apex-quant
• llama.cpp Engine: https://github.com/ggerganov/llama.cpp
• GGUF Specification: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md