Qwen3.6-35B-A3B-AntiLoop-NVFP4

Qwen3.6-35B-A3B-AntiLoop-NVFP4

This is a mixed-precision NVIDIA ModelOpt deployment checkpoint for Qwen3.6-35B-A3B-AntiLoop, a narrow fine-tune intended to recover from pathological self-verification and enumeration loops while preserving ordinary long-form reasoning.

The repository contains a complete deployment checkpoint; no PEFT adapter is required at inference time. It retains the upstream multimodal architecture, tokenizer, chat template, MTP weights, and 262,144-token native context length.

Training data

The exact 178 masked supervised targets used for the final AntiLoop training round are published in the Qwen3.6-35B-A3B-AntiLoop-SFT dataset. The dataset preserves each loss_start_char boundary so the pathological loop prefix remains conditioning context rather than a supervised target. It intentionally excludes the separately generated KL-regularization anchors.

Training procedure

The AntiLoop adapter was trained on the 178 masked supervised targets using a standard supervised fine-tuning procedure, but regularized via KL-loss on separately generated non-loop anchors from the base model on everyday prompts from tulu-3-sft-mixture.

Benchmark results

LoopHard

LoopHard is our held-out set of 285 enumeration prompts designed to elicit futile recall, recounting, and self-verification loops. The primary metric is judged loops: whether the model's reasoning trace remains stuck in a futile cycle when generation ends.

The matched presence_penalty=1.5 comparison converted all 30 control loops to clean completions while introducing one different loop. Exact two-sided McNemar p = 2.98e-8.

Generation used thinking mode, temperature=0.7, top_p=0.95, top_k=20, a 6,144-token completion limit, and concurrency 24. The two presence_penalty=1.5 arms used the exact original and AntiLoop NVFP4 checkpoints with the same vLLM build and serving configuration: TP1, FP8 KV cache, FlashInfer attention, Marlin NVFP4 MoE, and MTP speculative decoding with three draft tokens.

LoopHard is judged by GLM-5.2 using a convergence-aware rubric: systematic reasoning and verification that reaches a conclusion are not loops, and a trace that notices its own circling and exits is classified as recovered. The calibration set contained 42 manually labeled traces. Across three judge runs, accuracy was 88.1%, 92.9%, and 95.2%; all three runs identified all 17 labeled loops, with 2–5 false positives among the 25 non-loop traces.

The 285 prompts, metadata, and GLM-5.2 evaluation code are published in the LoopHard dataset on Hugging Face.

Capability preservation

The capability checks below compare the same round-2 AntiLoop adapter against its FP8 reference model under a matched runtime-LoRA setup. These runs used the default presence penalty and should not be interpreted as evaluations of the exact mixed-precision artifact at presence_penalty=1.5.

GPQA Diamond

The official-model-card number is included for context and was not produced by our harness.

Our GPQA run used thinking mode, paired per-question seeds, temperature=0.7, top_p=0.95, top_k=20, MTP3 speculative decoding, a 65,536-token reasoning budget, and 4,096 tokens of answer headroom. The difference was not significant.

Source for the published 86.0% result: Qwen/Qwen3.6-35B-A3B model card.

GSM8K

The GSM8K run used the exact 1,319-example openai/gsm8k main test split, thinking mode, paired seeds, temperature=0.7, top_p=0.95, top_k=20, MTP3, an 8,192-token reasoning budget and 1,024 tokens of answer headroom. The difference was not significant.

Taken together, the matched GPQA and GSM8K results show no material or statistically detectable capability loss at these sample sizes. They do not establish equivalence across other tasks, modalities, or sampling settings.

Usage

Use a recent vLLM build with ModelOpt mixed-precision support:

vllm serve N8Programs/Qwen3.6-35B-A3B-AntiLoop-NVFP4 \
  --quantization modelopt \
  --trust-remote-code \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3

MTP speculative decoding can be enabled on a compatible build with:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For the measured LoopHard setting, send the following sampling parameters:

{
  "temperature": 0.7,
  "top_p": 0.95,
  "top_k": 20,
  "presence_penalty": 1.5
}

The capability-preservation results above used the default presence penalty; presence_penalty=1.5 has not yet been evaluated on GPQA or GSM8K.

Follow the nvidia/Qwen3.6-35B-A3B-NVFP4 model card for deployment requirements and the Qwen/Qwen3.6-35B-A3B model card for chat templating, thinking controls, multimodal inputs, and base-model limitations.

Limitations

• This is a narrow behavioral fine-tune, not a general alignment or safety model.
• LoopHard is a task-specific, judge-based benchmark; its loop rate should not be interpreted as a general safety, truthfulness, or factuality score.
• The GPQA and GSM8K checks used runtime LoRA on an FP8 base, not this exact mixed-precision artifact.
• Capability preservation has not been tested at presence_penalty=1.5.
• Fixed-scale FP8 re-quantization approximates the exact BF16 LoRA merge; small adapter updates can round away or clip at the original E4M3 range.
• Runtime validation used a 65,536-token configured context, not the full native 262,144-token context.
• Multimodal generation quality has not been evaluated on this artifact.
• Outputs may still be incorrect, overconfident, repetitive, biased, toxic, or unsafe.

License

Apache 2.0, following both the underlying Qwen checkpoint and NVIDIA's quantized derivative. See the pinned Qwen license, the Qwen model card, and the NVIDIA ModelOpt checkpoint card.

(co-written with GPT-5.6-Sol)