Downloads · 30 days
0
oaica/oaica-35b-a3b-awq-mtp-vision-260830-enc
oaica-35b-a3b-awq-mtp-vision-260830-enc is a machine learning model from oaica. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
KAT-Coder-V2.5-Dev (Qwen3.5-35B-A3B MoE arch) AWQ W4A16 (compressed-tensors pack-quantized, group 128, asymmetric) text backbone + bf16 vision tower + bf16 MTP head (Qwen3.6-35B-A3B mtp., 1 layer) grafted for vLLM spe…
Downloads · 30 days
0
Access
Public
Updated Aug 31, 2026
Repo size
24.3 GB
Likes
0
Public
Click a slice to open those files.
.age24.3 GB · 100%
From the Hugging Face model README
KAT-Coder-V2.5-Dev (Qwen3.5-35B-A3B MoE arch) AWQ W4A16 (compressed-tensors pack-quantized, group 128, asymmetric) text backbone
mtp.*, 1 layer) grafted for vLLM speculative decoding.
Built by symlink-graft (no requant): AWQ shards from sprappcom/oaica-35b-malay-coder-awq-260827 lineage, model-mtp.safetensors bf16,
quantization_config.ignore includes re:.*mtp.* (required — otherwise vLLM's MTP loader raises KeyError: 'layers.0.mlp.experts.routed_experts.w2_weight', vllm#36954 class),
text_config.mtp_num_hidden_layers = 1.--kv-cache-dtype auto when MTP is on.VLLM_DISABLED_KERNELS=HummingLinearKernel python3 -m vllm.entrypoints.openai.api_server --model <dir> \
--max-model-len 8192 --gdn-prefill-backend triton --enable-prefix-caching --gpu-memory-utilization 0.90 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'
--gdn-prefill-backend triton is mandatory (FlashInfer GDN prefill deadlocks). Loads via Marlin (CompressedTensorsWNA16MarlinMoEMethod).
Do not use --kv-cache-dtype fp8 with MTP on Ampere — fp8 KV halves the MTP verify speed (N=1 103 vs 171 tok/s at 256k cfg).
| config | N=1 | N=32 agg (per-user) | N=64 agg | N=128 agg | N@>=30/u | accept len |
|---|---|---|---|---|---|---|
| AWQ + MTP(2) | 198.5 | 2936 (99) | 3902 (66) | 4861 | 166 | 2.26 |
| AWQ plain | 145.2 | 2095 (79) | 3668 (60) | 5026 | ~180 | – |
Greedy outputs are identical with and without MTP. VRAM 73.7 GB at util 0.90. Vision tower included (bf16); text-only benchmarked.
Crossover: MTP wins per-user and aggregate up to ~N=96; plain AWQ wins aggregate at N>=128.
Every *.age file in this repo is age-encrypted (recipient key: age19qgcqq4snawr6gz3cs8echcra98pyxgck2yvsrj7a40383vlcyvsp9vk77). Decrypt: age -d -i key.txt file.age > file. Integrity: decrypt SHASUMS.age to SHA256SUMS, then sha256 each decrypted file and compare. Private repo - useless without the age private key held in the operator secrets store.