Anti-DEGR-Toolkit

Anti-DEGR ToolKit

Training and decoding against four failures small LMs actually hit. I built it on Byrne / Escarda (86M SpikeWhale). It now loads any HuggingFace causal model that will take position_ids during cached decode: Length-MAX tokenizer.json stays on the SpikeWhale path, everything else falls through to AutoTokenizer.

D and R look the same if you only read the output. They are different bugs. D has a period. R is "Carrots can also be used as a substitute for regular carrots" followed by the same sentence with potatoes. I shipped 0% loop numbers that a person would still call repetitive because I was only measuring D.

I train E and G. I leave most of D to the sampler. Training loops out of the weights costs perplexity and I never got 0% that way without wrecking grounding. R is its own path: measure, try DRY, train only if a gap remains or if you export GGUF / WebGPU and the Python sampler never runs.

Do not serve these checkpoints greedy. You will see the loop rate I refused to buy with weights.

Install

git clone https://huggingface.co/Quazim0t0/Anti-DEGR-Toolkit
cd Anti-DEGR-Toolkit
pip install -r requirements.txt

--model is a HuggingFace id or a local folder. Drop a heldout.txt next to the scripts. I used WikiText-2 test. Any utf-8 text file works; anchors and ppl both read it.

Usage — my models

SpikeWhale checkpoints (Byrne, Escarda, Byrne-100M-Ultra-MC, the JL and Anti-DEG weight dumps) load as-is. ChatML is already how they were SFT'd. Use the base, not the chat stage, if you want to train: SFT is a thin probe on those releases.

# Byrne-86M base
python antideg.py --model Quazim0t0/Byrne-86M-Base --tag byrne86 --device cuda \
  --rank 16 --epochs 3 --loop-rounds 1 --echo-rounds 20

python effort_small.py --model adeg_byrne86_merged --levels low,high --topic-term --device cuda

# Escarda-86M base
python antideg.py --model Quazim0t0/Escarda-86M-Base --tag escarda86 --device cuda \
  --rank 16 --epochs 3 --loop-rounds 1 --echo-rounds 20

# Byrne-100M-Ultra-MC base (local .pt releases still want the inference folder
# that ships tokenizer.json + model_v2.py). Point at that folder, not the .pt.
python antideg.py --model I:/standalone/Byrne100M-Ultra-MC --tag ultra114 --device cuda \
  --rank 16 --epochs 3 --loop-rounds 1 --echo-rounds 20

Already-trained Anti-DEG weights I published:

• Quazim0t0/Byrne-86M-Base-JL-Anti-DEG
• Quazim0t0/Escarda-86M-Base-JL-Anti-DEG
• Quazim0t0/Escarda-86M-Identity-Anti-DEG
• Quazim0t0/Positronic-144M-Anti-DEG

Those still need stage B at serve:

python effort_small.py --model Quazim0t0/Byrne-86M-Base-JL-Anti-DEG \
  --levels high --topic-term --device cuda

Usage — any HuggingFace causal model

Same commands. --model can be any hub id or local transformers folder.

python antideg.py --model HuggingFaceTB/SmolLM2-135M --tag smol135 --device cuda \
  --rank 16 --epochs 3 --loop-rounds 1 --echo-rounds 20

python effort_small.py --model adeg_smol135_merged --levels low,high --topic-term --device cuda

What changes under the hood:

What does not change:

• Harvest still frames echo as ChatML (<|im_start|>user …). That is text. A Llama/Qwen/SmolLM that never saw those tokens still sees a subject change. If you want its native chat template, swap the strings in anti_echo.py / topic_switch.py before you harvest.
• Cached decode still passes position_ids. If a model rejects that kwarg, it is not a drop-in.
• Stage B is still not optional. Greedy eval of a merged adapter is the loop rate I did not train away.

Worked example, one failure at a time, if you do not want the driver:

# E — topic-switch pairs
python anti_echo.py --model <model> --out echo.jsonl --device cuda --rounds 20

# G — induction anchors on heldout.txt
python ga_ftpo.py anchors2 --model <model> --out anchors.jsonl --want 1200 --corpus heldout.txt --device cuda

# D — modest loop harvest (keep this small)
python small_antidoom.py harvest --model <model> --out loops.jsonl --device cuda --rounds 1 --max-new 300

# train E+G (GA-FTPO v2), then merge
python ga_ftpo.py train2 --model <model> --pairs pairs.jsonl --anchors anchors.jsonl \
  --adapter adapter --device cuda --rank 16 --lambda-mse 1.5 --lambda-ground 6.0 --margin 1.0
python merge_model.py --base <model> --adapter adapter --out merged --device cpu

# B — effort decode (this is what kills D)
python effort_small.py --model merged --levels low,high --topic-term --device cuda

antideg.py is those steps in order, plus echo oversampling into pairs.jsonl.

Near-copies (R), no model required to measure

from near_repetition import rep_rate, distinct_n, report
rep_rate(text)        # fraction of sentences that near-copy an earlier one
distinct_n(text, 3)   # unique 3-grams / total
report(text)

A sentence counts as a near-copy when max(unigram Jaccard, 4-gram Jaccard) against any earlier sentence clears thresh. DRY (near_repetition.dry_penalty) subtracts multiplier * base**(match_len - allowed_len) from a token that would continue a repeat. Flat rp=1.15 treats "this word appeared once" the same as "40 tokens into replaying a sentence", which is why loops survive it.

anti_repeat.py is the FTPO harvest if weights have to carry the fix (GGUF / WebGPU). Same pair schema as echo.

Checks I actually run after a merge

python ppl_check.py --model <base> --adapter adapter --corpus heldout.txt --device cuda
python topic_switch.py --models base=<base>,merged=<merged> --device cuda
python near_repetition.py   # or report() on dumped generations

What the driver does

python antideg.py --model <model> --tag <name> --device cuda

1. anti_echo.py — contexts where the subject changes and the model stays on the old one. Rejected token is the opener of that reply. Chosen prefers words from the new question. Training bank is disjoint from topic_switch.py.
2. ga_ftpo.py anchors2 — induction anchors: the (n−1)-gram before i already appeared and was followed there by token i.
3. small_antidoom.py harvest — a modest loop set on purpose.
4. ga_ftpo.py train2 then merge_model.py. Merge aborts if logits drift.

Decode

python effort_small.py --model <merged> --levels low,high --topic-term --device cuda

Score is Glint-2's:

mean_logprob - 2.0 * rep_frac - 1.5 * shortfall + topic_adherence

rep_frac is 4-gram overlap in the continuation, not find_inner_repetition. I did that on purpose so the reported loop rate is not the thing being maximised.

Levels: low / medium / high / xhigh / max / ultra. high is what I serve. max / ultra are extra independent runs of high.

Losses

FTPO is from Antidoom. One position. Forward the context up to the bad token, push logit[chosen] − logit[rejected] past a margin with softplus, decay the weight to zero once the margin is met, MSE-tether every other logit to the reference (same model, LoRA disabled).

GA-FTPO is the extra term I added:

L = L_pref + lambda_mse * tether
    + lambda_ground * softplus(margin - (logit[g] - max_{k != g} logit[k]))

v1 used softplus(ref[g] − logit[g]). It saturated (shortfall −6.9, ground 0.012) and the model could still change which token wins. Best loop number I ever got though (8%). v2 holds the argmax by a margin and stays live (ground 1.7–3.6). Inflating all logits together does not satisfy it.

Tuning

lambda_ground has to track the loop-pair : anchor ratio. At 2.0 with 303 loops vs 1200 anchors, win hit 84% and grounding lost. Same setting on 117 pairs was fine. If lead goes negative and kept falls, raise it.

Rank is capacity, not a quality knob. Three objectives at rank 16: win 65%, kept 25%, lead −0.12. Rank 64: 95% / 75% / +1.94. Two objectives at rank 16: 68% / 75% / +1.70. They were fighting over adapter slots, not fighting each other. Check the log for starvation before you touch loss weights.

Rank 64 also cost more ppl (+12.7% / +23.9%) than rank 16 (+0.2% / +15.0%). I only raise it when something is capacity-bound. Anti-DEGR stays at 16 because decoding owns loops.

Results

Byrne-86M. Loops over 90 generations, topic over 6 held-out switches.

Escarda-86M: 90% loops / 0.33 topic → 0% / 1.00, ppl +6.0%. First time I ran it with a fat loop set it cost +15.0%. Cutting the loop harvest is what dropped the bill.

The deciding ablation, 6 held-out topic switches:

Loops died at decode time on the untrained base. Every training run I did against loops paid between +4% and +37% perplexity and still left 10–46%. Echo did not move with more search. Decoding plateaued at 0.61; training is what got it to 1.00. Grounding is not a third trick you add. It is what the other two destroy if you ignore it.

Stuff I would not run again:

• GA on top of anti-echo. Loops 46% → 16%, topic 0.82 → 0.46, ppl +25.5%. The MSE tether pins logit values, not behaviour.
• GA-FTPO v2 at Byrne's hyperparameters on Escarda. Beaten on every axis by plain lambda_mse=3.0.
• Iterating λ=1.0 until 10% loops. Grounding went to 0.004, same as λ=0.4. Every training run that hit ~10% loops also killed copy rate.
• Using history 4-gram copy as a grounding metric. λ=3.0 scored 0.097 and was no better than base at topic switching. A lot of what it "kept" was parroting the last answer.

Things that bit me

The cycle detector missed a sample with longest_repeat=189 because slight alternation broke the period match. On the Mark2 DPO checkpoints it missed everything. Read rep_rate and the actual generations.

Echo rate is length-dependent. Same 8Final checkpoint, same held-out bank: 7.0% at 80 new tokens, 15.0% at 1000. Harvest at one length and eval at another and you have measured two different failures. I did this on 8Final/JEPA6.

Score ppl on a real corpus. I published +21.8% off concatenated prompt stems. On wikitext-2 test it was +15.4%.

This tokenizer (Length-MAX) is greedy longest-match, so tokenize(a+b) is not tokenize(a)+tokenize(b). Compare on shared ids or you will compare different sequences. That one made a result look violated at |ΔNLL| 0.27.

SpikeWhaleLM takes an additive 4D attention mask. A 2D binary mask trains on corrupted attention and does not error. selfcheck asserts batched logits match unbatched. SFT checkpoints with the engram open fail it; use --bucket so batches never pad. Other HF models get a 2D pad mask.

PeftModel.from_pretrained injects LoRA in place. Keep a handle to the pre-wrap model and you have the adapted model twice. disable_adapter() is the switch. ppl_check.py does this.

Layout

Credit

FTPO and the original cycle detector: Antidoom, Liquid AI, Sam Paech, Apache 2.0. Antidoom adapts Antislop.

Effort ladder, repetition-penalised sampler, and the logprob / rep / shortfall score: Glint-2, MIT.

GA-FTPO, anti-echo, the R path, and the pipeline: me, Dean Byrne (Quazim0t0).