erebus-v2-1.5b-base

Erebus v2 1.5B — Base

A 1.5B parameter causal language model pretrained from scratch using the Qwen3 architecture. This is the base (pretrained) model — see the instruct and tool-calling variants for fine-tuned versions.

Model Details

Training

• Hardware: 4x NVIDIA A100-SXM4-80GB
• Framework: PyTorch + Accelerate + Flash Attention 2
• Optimizer: AdamW (fused), β1=0.9, β2=0.95, ε=1e-8
• Learning rate: 4e-4 peak, WSD schedule (warmup → stable → decay)
• Warmup: 150 steps (linear)
• Batch size: 8 per device × 4 GPUs × 32 gradient accumulation = 1,024 sequences
• Sequence length: 2,048 tokens (packed)
• Weight decay: 0.1
• Max grad norm: 1.0
• Total steps: 2,644

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "soyrsoyr/erebus-v2-1.5b-base",
    torch_dtype="bfloat16",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("soyrsoyr/erebus-v2-1.5b-base")

inputs = tokenizer("The future of AI is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Variants

Limitations

This is a small (1.5B) research model trained on a limited token budget (5.5B tokens). It is not intended for production use. Outputs may be incoherent, repetitive, or factually incorrect. The fine-tuned variants (instruct, tool) are significantly more useful for downstream tasks.

License

Apache 2.0