AGILLM-3-large-v2

AGILLM-3-Large v2

Continuation of AGILLM-3-Large training, restarted from a known-good checkpoint after a critical tokenizer bug was discovered.

What happened to v1?

A transformers library update (to v5.3.0) on 2026-03-11 silently broke the DeepSeek-V3.2 tokenizer's encode/decode pipeline:

• Root cause: The tokenizer's Metaspace pre-tokenizer was configured to use ▁ (U+2581, SentencePiece convention) for space replacement, but the BPE vocabulary uses Ġ (U+0120, GPT-2 convention). This mismatch caused:

  • Encoding: All spaces were silently dropped. "Water boils" encoded to ['Water', 'bo', 'ils'] instead of ['ĠWater', 'Ġboils']
  • Decoding: tok.decode() lost all spaces. Round-trip encode→decode of "The meaning of life" returned "Themeaningoflife"
  • Training data corruption: ~3 billion tokens of training data were fed to the model without any space information, causing the model weights to degrade

• Detection: Model output went from coherent English (step 12,373,125) to space-less garbled text (step 12,528,061+). Took investigation to trace back to the tokenizer library update.

• Fix: Pinned transformers==4.48.0 (+ tokenizers==0.21.4), which correctly handles the Ġ space prefix. Also added a runtime fix in n.py that patches the ▁→Ġ mismatch if detected.

This repo

Resumes training from step 12,373,125 (~10.83B tokens, 30.9%) — the last checkpoint with correctly-encoded training data.

Model

Training setup

• GPU: RTX 4090 (Vast.ai, ~$0.27/hr)
• Speed: ~20,000 tok/s
• Block size: 1122
• Batch size: 1
• Mixed precision: AMP (BF16)
• Optimizer: AdamW (LR core=5e-5, LR head=2e-4)
• Data: Streamed from multiple HuggingFace datasets (web crawl, cleaned text)

Important: Tokenizer compatibility

This model requires transformers<=4.48.0 for correct tokenizer behavior:

pip install transformers==4.48.0 tokenizers==0.21.4

Or use the runtime fix in n.py which auto-patches the ▁/Ġ mismatch.

Sample output (step 12,373,125)

Prompt: "Water boils at one hundred degrees"

Water boils at one hundred degrees in the 1990s, and a year after that. "It's not just that, but it makes you think." He said: "I don't think I'm going to make a deal for the rest of my life."

Inference examples

This repository currently stores raw PyTorch training checkpoints and delta checkpoints. It is not a drop-in transformers.pipeline() model yet. Use the AGILLM training/inference script (nB300.py) with the pinned tokenizer stack above.

Download the newest checkpoint artifact

from huggingface_hub import hf_hub_download, list_repo_files

repo_id = "MarxistLeninist/AGILLM-3-large-v2"

def step_number(path: str) -> int:
    return int(path.rsplit("step", 1)[1].split(".", 1)[0])

checkpoint_files = [
    path for path in list_repo_files(repo_id)
    if path.startswith("pretrain_delta_step") and path.endswith(".pt")
]
latest = max(checkpoint_files, key=step_number)
ckpt_path = hf_hub_download(repo_id=repo_id, filename=latest)
print(ckpt_path)

If a full pretrain_step*.pt checkpoint is available, prefer it for optimizer-preserving resumes. Delta checkpoints are weight-only and are fine for text generation.

Greedy factual completion

python nB300.py infer \
  --mode ar \
  --ckpt ./pretrain_delta_step27775678.pt \
  --prompt "Water boils at one hundred degrees" \
  --max_new 120 \
  --greedy \
  --plain-output

Nucleus-sampled continuation

python nB300.py infer \
  --mode ar \
  --ckpt ./pretrain_delta_step27775678.pt \
  --prompt "The history of machine learning began with" \
  --max_new 180 \
  --temperature 0.7 \
  --top_p 0.9 \
  --repetition_penalty 1.3 \
  --frequency_penalty 0.3 \
  --plain-output

SAT mode creative continuation

python nB300.py infer \
  --mode sat \
  --ckpt ./pretrain_delta_step27775678.pt \
  --prompt "In a quiet orbital station above Jupiter," \
  --max_new 180 \
  --temperature 0.5 \
  --top_k 30 \
  --top_p 0.9 \
  --presence_penalty 0.6 \
  --frequency_penalty 1.0 \
  --plain-output

Prompt examples to try

The model is still in pretraining, so outputs should be treated as experimental continuations rather than instruction-tuned answers.

License

Apache 2.0