Ornith-1.0-35B-AWQ-asym

Ornith-1.0-35B-AWQ-asym (for 1Cat-vLLM on V100 / sm_70)

Asymmetric AWQ W4A16 (group-size 128, zero-point) quantization of deepreinforce-ai/Ornith-1.0-35B, produced specifically to run on NVIDIA V100 (sm_70) GPUs via 1Cat-vLLM.

⚠️ NOT YET VALIDATED

Quantization quality has not been benchmarked or validated for accuracy. It loads and generates coherent output, but no eval/perplexity comparison vs the base model has been run. Published to avoid re-quantizing; validate before relying on it.

Why asymmetric AWQ specifically

Ornith-1.0-35B is a qwen3_5_moe (256-expert fine-grained MoE, GDN hybrid attention). On V100/sm_70, 1Cat-vLLM'''s awq_sm70_moe (TurboMind) kernel is the only working 4-bit MoE path — and it requires asymmetric AWQ (zero points). Every other published format (compressed-tensors INT4/FP8, GPTQ/AutoRound-gptq, NVFP4, and even symmetric AWQ) routes to Marlin kernels that have no sm_70 build and fail to load.

How it was made

auto-round-mllm --model deepreinforce-ai/Ornith-1.0-35B \
  --scheme W4A16 --algorithm awq --asym --format auto_awq \
  --output_dir <out>

Activation-aware AWQ, calibrated on NeelNanda/pile-10k. (An RTN variant was also made.)

Deploy (1Cat-vLLM, V100, TP1 — one card fits the 20 GB weights)

vllm serve <this-model> --quantization awq --trust-remote-code --dtype float16 \
  --max-model-len 262144 --kv-cache-dtype fp8_e5m2
# env: VLLM_ATTENTION_BACKEND=FLASH_ATTN_V100

Scales by data-parallel replication (each TP1 replica ~97 tok/s single-stream); the fine-grained MoE batches poorly, so prefer many single-stream replicas.