AgentWorld-35B-A3B-Heretic-FP8

AgentWorld-35B-A3B-Heretic — FP8

Static FP8 quantization of auttasak88/AgentWorld-35B-A3B-Heretic in compressed-tensors format, for efficient serving with vLLM.

• ~34 GB (vs 66 GB BF16) — runs on 2 GPUs instead of 4.
• FP8 (E4M3) weights + dynamic FP8 activations (FP8_DYNAMIC), including the 256 fused MoE experts and full-attention / shared-expert layers. The GatedDeltaNet linear-attention layers, MoE routers, and lm_head are kept in higher precision.
• Native FP8 speedup on Ada / Hopper / Blackwell (RTX 4090 / H100 / RTX 5090); on Ampere (RTX 3090) it runs weight-only (VRAM savings, no compute speedup).
• Quantized with llm-compressor (data-free — no calibration set).

For the full model description, benchmarks, methodology, and other formats (BF16 safetensors, GGUF Q4/Q5/Q8), see the base model: auttasak88/AgentWorld-35B-A3B-Heretic.

This is a capability-preserving abliteration of Qwen/Qwen-AgentWorld-35B-A3B (a language world model for agent–environment interaction) produced with Heretic: refusal / over-refusal behavior is substantially reduced while coding, reasoning, multilingual (incl. Thai), and agent-environment modeling are largely preserved. See the base card for the evidence and caveats.

Usage (vLLM)

The checkpoint is language-model-only (no vision weights). vLLM detects the compressed-tensors FP8 config automatically.

vllm serve auttasak88/AgentWorld-35B-A3B-Heretic-FP8 \
  --tensor-parallel-size 2 --language-model-only \
  --max-model-len 8192 --max-num-seqs 64

• Keep --max-num-seqs ≤ ~120 — the linear-attention (GatedDeltaNet) layers use a Mamba-style cache, and a too-high value fails CUDA-graph capture (max_num_seqs exceeds Mamba cache blocks).
• On Ampere, if the FP8 GEMM kernel needs runtime compilation, add the CUDA nvrtc lib dir to LD_LIBRARY_PATH, or force the Marlin FP8 kernel with VLLM_TEST_FORCE_FP8_MARLIN=1.

License & attribution

Apache-2.0, inheriting the base model. Abliteration by Heretic; FP8 by llm-compressor; base model by the Qwen team (Qwen/Qwen-AgentWorld-35B-A3B).