Qwen3.5-35B-A3B-MXFP8-FP8KV-CT-AutoRound

Qwen3.5-35B-A3B-MXFP8-FP8KV-CT-RTN-AutoRound

Model Details

This is a MXFP8 quantization of Qwen/Qwen3.5-35B-A3B, generated by AutoRound. Please follow the license of the original model.

Quantization Details

• Scheme: MXFP8
• Method: AutoRound
• Base model: Qwen/Qwen3.5-35B-A3B

Evaluation

Reproduce & Environment

Quantized with:

auto-round --model_name /models/Qwen3.5-35B-A3B --scheme MXFP8 --iters 0 --static_kv_dtype fp8 --ignore_layers lm_head,embed_tokens,visual,in_proj_a,in_proj_b,conv1d,shared_expert_gate,mlp.gate --format llm_compressor --output_dir results/qwen3_5_35b_a3b/mxfp8_v2/quantized --disable_low_cpu_mem_usage

Evaluated with:

lm_eval --model vllm --model_args pretrained=results/qwen3_5_35b_a3b/mxfp8_v2/quantized/Qwen3.5-35B-A3B-mxfp-w8g32,tensor_parallel_size=1,max_model_len=8192,gpu_memory_utilization=0.3,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,max_num_batched_tokens=32768,max_num_seqs=128,enable_thinking=False,reasoning_parser=qwen3,safetensors_load_strategy=prefetch,kv_cache_dtype=fp8 --tasks gsm8k --batch_size 64 --output_path results/qwen3_5_35b_a3b/mxfp8_v2/eval/gsm8k --log_samples --seed 42 --num_fewshot 5 --apply_chat_template --fewshot_as_multiturn
lm_eval --model vllm --model_args pretrained=results/qwen3_5_35b_a3b/mxfp8_v2/quantized/Qwen3.5-35B-A3B-mxfp-w8g32,tensor_parallel_size=1,max_model_len=8192,gpu_memory_utilization=0.3,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,max_num_batched_tokens=32768,max_num_seqs=128,enable_thinking=False,reasoning_parser=qwen3,safetensors_load_strategy=prefetch,kv_cache_dtype=fp8 --tasks piqa,mmlu,hellaswag --batch_size 64 --output_path results/qwen3_5_35b_a3b/mxfp8_v2/eval/piqa_mmlu_hellaswag --log_samples --seed 42

Inference: load this repo with a vllm build that supports the MXFP8 format (day-0 architectures may need the model card's custom image/branch — check the quantize command above for the environment used).

Produced with autoquant-agent — agent-driven quantize + evaluate + self-heal.