LiLM1-230M-base

LiLM1-230M-base

LiLM1-230M-base is the pretrained LiLM1 checkpoint. It has 233,897,728 parameters and a 4,096-token context. The public name uses the nearest 10M parameter class.

This is a base model. It is not instruction-tuned.

Model details

The hybrid stack uses width 1,024, SwiGLU intermediate size 2,560, and tied input and output embeddings. Convolution layers use kernel size 3. Attention layers sit at depths 2, 5, 8, 11, 13, and 15. Attention uses 16 query heads, 4 key/value heads, head dimension 64, and Q/K RMSNorm. Position encoding is RoPE with base theta 10,000.

Training

The base model used 40B token exposures at 4,096 context. Training used BF16 on one NVIDIA RTX PRO 6000 Blackwell GPU. The recorded steady rate was 147K to 148K tokens per second. This gives a standardized pretraining estimate of 75.1 to 75.6 GPU-hours.

The public pretraining corpus is not available.

Evaluation

The 40B base checkpoint scores 45.22% on an 11-task zero-shot aggregate. WikiText word perplexity is 26.71. WikiText bits per byte is 0.886.

Same-scale public references remain ahead on this aggregate. SmolLM2 135M scores 47.84%. LFM2.5 230M scores 48.73%. SmolLM2 360M scores 54.02%. LiLM1 used 50 times fewer pretraining tokens than SmolLM2 135M.

The base checkpoint is a next-token model. It is not a tool-use or instruction-following model.

Load

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "glouriousgautam/LiLM1-230M-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)

The architecture is custom. Loading requires trust_remote_code=True.

Intended use

Use LiLM1-230M-base for research on compact language models. It can serve as the parent for further training. Validate any downstream use before deployment.

Do not use this checkpoint as a chat or tool-calling model. Do not use it with context longer than 4,096 tokens. Do not use it for medical, legal, financial, or security decisions.

Limitations

• English is the only evaluated language.
• The model has limited math, retrieval, and instruction-following ability.
• The evaluation does not cover every downstream task.
• No quantized release has passed final validation.

License

Apache License 2.0.

Citation

@misc{lilm1_base_2026,
  title        = {LiLM1-230M-base},
  author       = {Akshay Gautam},
  year         = {2026},
  howpublished = {https://huggingface.co/glouriousgautam/LiLM1-230M-base}
}