Nemotron-3.5-30B-A3B-Antislop-FTPO-V2

Nemotron-3.5-30B-A3B-Antislop-FTPO-V2

A merged, ready-to-serve BF16 checkpoint of NVIDIA's Nemotron 3.5 30B-A3B, tuned with Antislop and FTPO to suppress the overused lexical patterns that make LLM prose recognisable as LLM prose.

43.2% of overused patterns suppressed across six domains, 67.8% on creative writing, with no capability regression on any benchmark measured.

The LoRA adapter that produced this checkpoint is released separately at thoughtworks/Nemotron-3.5-30B-A3B-Antislop-FTPO-V2-LoRA.

What changed from V1

V1 suppressed 66.41% on creative writing prompts and was measured only there. V2 keeps the method and widens the scope:

V1 and V2 headline numbers are not directly comparable. V1 scored against its own 4,267-pattern banlist; V2 scores against a banlist 1.9x larger. On V1's exact 388 creative prompt IDs, V2 measures 67.78% — level with V1 while also covering five more domains. V1's benchmarks were also run with thinking on and V2's with thinking off, so the absolute MMLU and IFEval figures differ between cards for protocol reasons, not model reasons.

What this is

Language models overuse a characteristic vocabulary: stock dialogue tags, a fixed set of atmospheric props, and a long tail of default names. Antislop identifies those patterns for this specific model by comparing its output against a human-written baseline, then uses a backtracking sampler to catch the model reaching for one and record what it should have reached for instead. Those records become preference pairs, and FTPO (Final Token Preference Optimization) trains the preference into the weights so it persists with the sampler switched off.

V2 adds a second axis: typography and punctuation. V1's n-gram analysis dropped every non-letter token before counting, so it was structurally blind to em dashes, ellipses, curly quotes, emoji, and markdown. V2 profiles 28 such features against human rates and fires bans proportionally — with probability p = 1 - human_rate / model_rate, so the residual lands at human parity rather than zero. Suppressing a tic to far below human usage is its own detectable signature.

Results

952 held-out prose prompts, identical sampling for both models (temp 1.0, top_p 1.0, top_k 50, min_p 0.01), Antislop sampler off. These numbers reflect what training changed in the weights, not what a sampler suppresses at inference.

No benchmark difference is meaningfully distinguishable from zero at these sample sizes. HumanEval+ is worth calling out: V2 deliberately removed all code and math prompts from the training pool, and code ability is unchanged (one problem out of 164).

Suppression by domain

Suppression is strongest where the slop is densest and the register is furthest from the model's assistant default. Conversational is the weak case at 4.65%.

Typography: a split result

This is the new capability in V2, and it works for some feature classes and fails for others. Rates per 100k prose chars against the human baseline, code stripped:

Improved:

Made worse:

The split is not random. Features with an exact substitute get fixed; features requiring a structural choice get worse. A curly apostrophe maps to ', so FTPO's "chosen" token is well defined and learnable. An em dash's replacement is a comma, a colon, a period, or a rewritten clause depending on the sentence — the chosen token is effectively arbitrary and the gradient carries no consistent signal. Markdown headers, bullets and emoji fail the same way.

If you need em dashes suppressed, use the Antislop sampler at inference. Proportional firing held them at ~1.0x human during generation; it is the weight-baking step that fails.

Note also that several of the wins overshoot to below human rates. Structural diagnostics moved toward human on five of six measures (sentence length mean/sd/cv, conjunction openers, single-sentence paragraphs) and away on one (paragraph length).

What gets banned

The banlist is not a hand-written style guide. It is 8,033 patterns measured as overused in this model's own output relative to human prose: 4,342 n-grams (2,218 bigrams, 2,124 trigrams) and 3,990 slop phrases.

Genuine prose slop dominates: fingers hovering, voice echoes, impossibly large, eyes wide terror, short sharp sound, bruised purple choked, critics argue, particular kind magic, chuckled sound like.

Known limitations

• Em dashes and markdown structure get worse, as above.
• The 43.21% figure understates true slop suppression. The banlist contains prompt echoes that were never slop: character names (elara, kaelen, huck, sansa), code tokens (pygame, cloudformation), and non-English function words. These inflate the denominator. A prompt-echo filter is the main outstanding fix.
• No writing-quality judge was run. V1 reported an LLM-judge comparison; V2's judge and τ-bench runs were blocked by an invalid API key and are simply absent, not null results.
• Conversational and explanatory registers barely move (4.65% and 10.13%).
• Suppression is a reduction, not a hard filter.

Usage

Requires trust_remote_code=True for the nemotron_h architecture.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "thoughtworks/Nemotron-3.5-30B-A3B-Antislop-FTPO-V2"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True
)

messages = [{"role": "user", "content": "Write the opening of a story about a lighthouse keeper."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))

Note: on transformers 5.14.x, apply_chat_template(..., tokenize=True) under the TokenizersBackend tokenizer class returns a truncated sequence. Render with tokenize=False and tokenize the resulting string, as shown above. This affects the upstream base model identically and is not specific to this checkpoint.

The chat template defaults to thinking on, emitting an open <think> block. Pass enable_thinking=False to apply_chat_template (or chat_template_kwargs={"enable_thinking": false} over an OpenAI-compatible API) for direct prose. All prose evaluation above was run with thinking off.

vLLM

Stable vLLM 0.27.1 and later support NemotronHForCausalLM natively.

vllm serve thoughtworks/Nemotron-3.5-30B-A3B-Antislop-FTPO-V2 \
    --trust-remote-code \
    --moe-backend auto \
    --mamba-cache-mode align \
    --enable-prefix-caching \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 64 \
    --max-model-len 65536

For tool calling, add --enable-auto-tool-choice --tool-call-parser qwen3_xml, which handles this model's XML tool syntax correctly.

llama.cpp

Supported from build b9610 (nemotron_h_moe). When quantizing, keep the output tensor at f16:

llama-quantize --output-tensor-type f16 model-bf16.gguf model-Q4_K_M.gguf Q4_K_M

The entire fine-tune lives in lm_head as a perturbation of mean 3.3e-05. A default Q4_K_M puts the output tensor at Q6_K, whose quantization noise is large enough to swamp a delta that small and quietly undo the training. Expect ~24.9 GB at 6.29 BPW rather than the usual ~19 GB, partly for this reason and partly because 134 of 401 tensors fall back to q5_0 (moe_intermediate_size 1856 is not a multiple of the Q4_K block size).

Training

Training-set filtering removed 11,900 pairs of rare content vocabulary (Zipf < 2.5, e.g. proper nouns mistaken for slop), 5,315 non-Latin-script pairs, and 2,089 displacement-only pairs where the model had merely rerouted one suppressed tic into another.

Model architecture

Inherited unchanged from the base model. A hybrid Latent Mixture-of-Experts design with interleaved Mamba-2 and MoE layers plus select attention layers, and Multi-Token Prediction (MTP) layers for speculative decoding. 52 layers (23 Mamba-2, 23 MoE, 6 attention), 128 experts with 6 active plus 1 shared, 30B total parameters, 3B active. Context length up to 262,144.

License

Released under OpenMDW-1.1, matching the license NVIDIA applies to the public Nemotron 3.5 Lightning releases. The Antislop framework itself is MIT-licensed.

Citation

The method was published at ICLR 2026:

@inproceedings{paech2026antislop,
  title     = {Antislop: A Comprehensive Framework for Identifying and Eliminating
               Repetitive Patterns in Language Models},
  author    = {Paech, Samuel and Roush, Allen and Goldfeder, Judah and Shwartz-Ziv, Ravid},
  booktitle = {The Fourteenth International Conference on Learning Representations},
  year      = {2026},
  url       = {https://openreview.net/forum?id=gLcyM1khyp},
  eprint    = {2510.15061},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL}
}

• Paper: ICLR 2026 poster · OpenReview · arXiv:2510.15061
• Code: github.com/sam-paech/auto-antislop (MIT)

Acknowledgements

Built on NVIDIA's Nemotron 3.5, with thanks to the NVIDIA team for the benchmark guidance that shaped the evaluation.