Nero-150M

Nero-150M

A 151,020,544-parameter decoder-only base language model trained from scratch for exactly 8 billion tokens.

Quick start

pip install "transformers>=5" torch safetensors

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "j0no12/Nero-150M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")

prompt = "Question: What is the capital of Japan?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=32,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Nero-150M is a base completion model, not a chat assistant. Prompt it with text to continue or an explicit question-and-answer format.

Results

All scores use zero-shot continuation likelihood on the complete evaluation split.

HellaSwag, ARC, and PIQA were evaluated with lm-eval in BF16 on Apple Metal using zero few-shot examples. ArithMark-3 used its official independent-tokenization protocol in FP32. Raw result JSON is included under evals/.

Architecture

Nero was implemented for training as NeroForCausalLM, a purpose-built PyTorch decoder. Its operations deliberately match the standard Hugging Face Llama graph, so this release loads without remote code as LlamaForCausalLM. “Llama-compatible” describes the interface and computation graph; the weights, tokenizer, dimensions, and training run were created from scratch.

Training story

• Data: 70% FineWeb-Edu sample-10BT, 30% DCLM baseline.
• Exposure: exactly 8,000,000,000 tokens.
• Schedule: 7B tokens at context 1,024, followed by a 1B-token cosine cooldown at context 2,048.
• Batch: 131,072 tokens globally and physically.
• Optimizer: AdamW, peak learning rate 6e-4, 200M-token warmup, minimum learning rate 10% of peak, weight decay 0.1, betas (0.9, 0.95), epsilon 1e-8, gradient clipping 1.0.
• Seed: 12,044 for training; 12,042 for tokenizer construction; 12,043 for corpus preparation.
• Compute: one Modal B200. The frozen preflight benchmark measured 817,950 tokens/s at 1K context and 752,121 tokens/s at 2K context.
• Numerics: FP32 master parameters, optimizer states, and reductions; BF16 autocast compute; BF16 release weights.

The production input pipeline resolved mutable dataset revisions before preprocessing, packed source data into hashed uint16 shards, and maintained resumable source cursors. The compact public snapshot contains the final model, tokenizer, exact token count, run fingerprint, and weight digest; it does not contain optimizer state or the private prepared corpus.

Evaluation and selection

The 8B-token base checkpoint was selected over the available SFT merge before release. On a fixed 100-example-per-task screen across HellaSwag, ARC-Easy, ARC-Challenge, and PIQA, base averaged 47.75% normalized accuracy versus 45.25% for SFT. The full benchmark table above was then run on the selected base checkpoint.

This selection used benchmark data, so the reported scores are descriptive release measurements rather than an untouched final estimate.

Reproducibility

The original export pipeline validated tokenizer size and special tokens, tied input/output embeddings, causal isolation, and logit parity between the native NeroForCausalLM implementation and the Transformers export. This release was independently reloaded with AutoModelForCausalLM; it reports 151,020,544 tied parameters.

Limitations

• This is a small base model. It is not instruction-tuned and should not be expected to behave like a chat assistant.
• Greedy or weakly penalized generation can repeat phrases. Use nucleus sampling and a modest repetition penalty.
• It can produce factual errors, flawed arithmetic, incoherent text, or unsafe content.
• English dominates the training mixture; multilingual ability was not established.
• No safety tuning, independent human evaluation, or comprehensive contamination audit is claimed.
• The public snapshot does not expose the resolved source revision ledger, so exact corpus reconstruction is not possible from this repository alone.

Files

Citation

@misc{nero150m2026,
  author       = {j0no12},
  title        = {Nero-150M: A 151M-Parameter Base Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/j0no12/Nero-150M}},
  note         = {Trained from scratch for 8 billion tokens}
}

License

The original Nero-150M weights and repository material are released under CC-BY-4.0. Attribution is required. Upstream dataset licenses and attribution requirements remain applicable to their respective source material.