Axion-1B-FineWebEdu-72K

Axion 1B FineWeb-Edu — Step 72K

This is an early base-model checkpoint from the Axion three-stage pipeline. It has 1,275,168,768 parameters and was pretrained on FineWeb-Edu with the Axion 16K SentencePiece tokenizer.

Checkpoint

• Optimizer step: 72,000
• Training tokens seen: 4,718,592,000
• Validation cross-entropy: 2.673866
• Validation perplexity: 14.49590
• Context length: 32,768 tokens
• Weight format: BF16 SafeTensors

This checkpoint is not instruction-tuned or chat-tuned. It may generate incorrect, repetitive, biased, or otherwise unreliable text.

Transformers

Custom model code is included because Axion is not a stock Llama architecture. Review the repository code before enabling trust_remote_code.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MyLabs-LLC/Axion-1B-FineWebEdu-72K"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="bfloat16",
    device_map="auto",
)

inputs = tokenizer("The future of science is", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))

vLLM

Axion uses vLLM's Transformers modeling backend:

vllm serve MyLabs-LLC/Axion-1B-FineWebEdu-72K \
  --trust-remote-code \
  --model-impl transformers \
  --dtype bfloat16 \
  --max-model-len 32768

For a 12GB GPU, begin with --max-model-len 8192 to leave sufficient room for the KV cache. A 24GB GPU can use the full context more comfortably.

Upload

From the Axion repository:

hf repos create MyLabs-LLC/Axion-1B-FineWebEdu-72K --private --exist-ok
hf upload-large-folder MyLabs-LLC/Axion-1B-FineWebEdu-72K hf_export/macho-1b-finewebedu-20b-step72000

No model license is asserted by this export. The repository owner should select and document a license before making the model public.