novai-base-102m

NovAI Base 102M

NovAI Base 102M is an Indonesian-language causal language model developed by KaryaVirtual.

Created by: Nova Novriansyah, MSc, MBA

Model Details

• Parameters: 102,558,240 (~102.6M)
• Architecture: Llama-style decoder-only Transformer
• Layers: 12
• Hidden size: 720
• Intermediate size: 1,920
• Attention heads: 12
• KV heads: 4
• Vocabulary size: 50,257
• Context length: 1,024 tokens
• Tokenizer: GPT-2 tokenizer

Pretraining

The model was initialized with newly created model weights and pretrained on approximately 2 billion Indonesian-language tokens.

Training hardware:

• NVIDIA A100 40GB

Dataset

The training corpus was prepared from HuggingFace FineWeb-2, using the Indonesian Latin (ind_Latn) subset.

Approximately:

• 2 billion training tokens
• 10 million validation tokens

The original dataset is not redistributed with this repository.

Users should refer to the upstream dataset documentation, license, and terms.

Source Repository

Technical documentation and supporting resources:

https://github.com/novrian6/novai

Training

The model was pretrained using causal language modeling.

• Training tokens: ~2 billion
• Validation tokens: ~10 million
• Training steps: 15,259

Full pretraining logs and training source code are available in the project repository: https://github.com/novrian6/novai

Important

This model is not a fine-tuned copy of Llama, GPT, Claude, Gemini, or another pretrained foundation model.

The architecture uses the Llama implementation available through Hugging Face Transformers, while the model weights were newly initialized and pretrained separately for NovAI.

Training Evidence

The documented training run completed at:

15,259 / 15,259 steps

Training loss:

1.7229

Validation loss:

1.7012

Approximate training tokens:

2,000,027,648

Tokenizer

The model uses the GPT-2 tokenizer with a vocabulary size of 50,257 tokens.

Tokenizer: https://huggingface.co/openai-community/gpt2

Intended Use

NovAI Base 102M is intended for:

• Indonesian NLP research
• language-model experimentation
• downstream fine-tuning
• local inference experimentation
• educational and research purposes

Limitations

NovAI Base 102M is a relatively small language model and should not be compared directly with substantially larger commercial or open-weight foundation models.

The model may generate incorrect, repetitive, biased, or factually unreliable content.

This repository contains the base pretrained model and is not an instruction-tuned conversational model.

License

The NovAI Base 102M model weights are released under the Apache License 2.0.

The training dataset, tokenizer, and third-party software remain subject to their respective licenses and terms.

Attribution

Model: NovAI Base 102M
Developer: KaryaVirtual
Author: Nova Novriansyah, MSc, MBA

Website: https://karyavirtual.com/