Clyrai_Lucius-1.3B-Base

<p align="center"> <img src="lucius-hero.png" alt="Clyrai Lucius — 1.3 billion parameters. One billion tokens." width="100%"> </p> <br>

Every parameter fires.

That is the difference.

Maximus is sparse. Four experts speak. The rest wait.

Lucius is dense. 1,342,277,632 weights. All of them move for every word. He is Clyrai's first language model — the heir, named for Gladiator II — walked into the arena before the house had enough grain.

Chinchilla asked for about twenty tokens per parameter. That is ~27 billion unique tokens. This checkpoint saw 1,000,341,504 training tokens from a 1.10B unique pack. One pass. No epoching. The ratio is the story.

This is the base. Not a chatbot. Not the effort-tagged fork. Completions only.

---

What he is

A Llama-style decoder trained from random initialization. RoPE. SwiGLU. GQA. 4,096 tokens of context. English, हिंदी, Italiano, code. Same tokenizer family as Maximus.

Held-out test CE 1.21 (PPL 3.35). Code is the easy bucket. English and Italian are the hungry ones. That is what 0.75 tokens per parameter looks like.

A later instruct experiment lives at Clyrai_Lucius-1.3B. Post-training can teach a turn. It cannot mint the pretrain that never happened. This repo is the engine.

---

Use it.

Give him a sentence that has already begun. Greedy decoding without a penalty will loop.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "saishshinde15/Clyrai_Lucius-1.3B-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
).cuda()

def complete(prompt: str) -> str:
    ids = tok(prompt, return_tensors="pt")
    ids = {k: v.to(model.device) for k, v in ids.items() if k in ("input_ids", "attention_mask")}
    out = model.generate(
        **ids,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.15,
        pad_token_id=tok.eos_token_id,
    )
    return tok.decode(out[0], skip_special_tokens=True)

print(complete("In a statement on Monday, the government"))
print(complete("भारत की राजधानी"))

Weights are 2.68 GiB bf16. CUDA. An 8 GB Air will not like this.

---

One more thing.

Lucius went first. The data was short. The architecture was dense. That is not a scandal. That is a first model.

The Mixture that followed — Maximus-MoE — was given the unique-token budget this dense 1.3B did not get. 110 million active. 5.2 billion unique. The heir taught the house what hunger looks like.

<p align="center"><br><em>Clyrai</em></p>

---

The work is ours.

The weights are public. The title is not.

Study it. Cite it. Do not take it. Research and personal evaluation, with credit. Commerce, distillation, hosted product use, and training another model on Lucius require a grant from Clyrai.

All other rights reserved.

Clyrai Sovereign Research License 1.0

---

<details> <summary>The numbers, for the curious</summary> <br> </details>