rigel-mid

Rigel is a 2.3B-total, 360M-active-parameter hybrid Mamba-2 + attention mixture-of-experts base model, trained with LM Engine. It used about 117× fewer pretraining FLOPs than Llama-3.2-1B/3B and 53× fewer than Granite-4.2-3B, while landing within a few points of Granite-4.2-3B and Llama-3.2-3B on zero-shot averages and above Llama-3.2-1B.

Read the full write-up, including architecture, data curriculum, the multi-chip training infrastructure, and evaluations: Rigel Base: Reaching Llama-3.2 Quality with <1% of its Compute.

This repo is the 294,912-token long-context extension, see open-lm-engine/rigel-base-long.

Model summary

This is a base model: it has no chat template and is not instruction-tuned.

Usage with LM Engine

Rigel's architecture lives in LM Engine. Installing it and importing lm_engine.training registers the architecture with transformers' Auto classes, after which the usual Hugging Face API works.

git clone https://github.com/open-lm-engine/lm-engine.git && cd lm-engine && uv sync --extra cuda

import torch
import lm_engine.training  # registers Rigel's architecture with transformers' Auto classes
from lm_engine.training.kernels import Kernel, enable_kernels
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "open-lm-engine/rigel-mid"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map={"": "cuda"}).eval()

inputs = tokenizer("Rigel is a blue supergiant star in the constellation", return_tensors="pt").to("cuda")
with torch.no_grad(), enable_kernels([Kernel.mamba2_ssm, Kernel.causal_conv1d, Kernel.sonicmoe, Kernel.flash_attention_3]):
    output_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Any kernel left out of the list falls back to a pure-PyTorch implementation, so the snippet also runs with an empty list, just slower.

To finetune or continue pretraining, point an LM Engine training config at this checkpoint; see the LM Engine README.

Citation

@misc{mishra2026rigel,
  title  = {Rigel Base: Reaching Llama-3.2 Quality with <1% of its Compute},
  author = {Mishra, Mayank and Runwal, Bharat and Stoica, Ion and Dao, Tri and Gonzalez, Joseph E.},
  year   = {2026},
  url    = {https://open-lm-engine.github.io/blog/rigel/}
}