Titanius-1-115m-base-fp16-preview

Titanius-1-115m-base-fp16-preview

Repository: NadavSalem/Titanius-1-115m-base-fp16-preview

Titanius-1 is an experimental decoder-only base language model. This preview is the result of a default training run with no command-line parameter overrides, stopped at the nominal 7,000-step checkpoint.

Preview status: This is a research artifact, not an instruction-tuned or production-ready assistant. It may generate inaccurate, biased, unsafe, or otherwise undesirable text.

Training overview

The token estimate is 7,000 × 65,536. No held-out validation or standardized benchmark evaluation was recorded for this preview, so training loss should not be interpreted as a measure of downstream quality.

Model architecture

The model uses RMS normalization in the forward path, QK normalization, untied input/output embeddings, per-layer residual scaling, an input-embedding residual, and a mid-layer “backout” residual.

Parameter count

The parameter count was recalculated directly from checkpoints/model_step_7000.pt:

The repository’s “115m” label is a rounded name for the approximately 113.25M active-parameter model. The raw training checkpoint is larger because every attention module instantiates an additional MLP and LayerNorm that are not called by the model’s forward pass. Those tensors are nevertheless included in the PyTorch state dictionary. Consumers exporting or converting the model should account for this distinction.

Tokenizer

The model uses a custom byte-level BPE tokenizer with a vocabulary of 32,768 tokens. Numbers are pre-tokenized in groups of up to three digits. Its special tokens are:

<|bos|>
<|user_start|>
<|user_end|>
<|assistant_start|>
<|assistant_end|>

Although conversation boundary tokens exist in the vocabulary, this checkpoint is a base pretrained model and has not been instruction-tuned.

Intended use

This checkpoint is intended for:

• research and experimentation with small causal language models;
• continued pretraining or supervised fine-tuning;
• inspection of the custom architecture and training run;
• non-critical text-generation experiments.

It is not intended for autonomous decisions, high-stakes applications, factual lookup, or deployment without application-specific evaluation and safeguards.

Limitations

• This is an early preview trained for only 7,000 steps (about 459M tokens).
• No benchmark, validation-set, safety, bias, or memorization evaluation is reported.
• The model is not instruction-tuned and may not reliably follow prompts.
• Outputs may be incorrect, incoherent, biased, offensive, or repetitive.
• The training corpus may contain problematic, copyrighted, personal, or low-quality material inherited from its upstream sources.
• Loading and inference require the repository’s custom PyTorch model code; this is not a drop-in standard Transformers architecture.

Training data

The run streams the training split of karpathy/climbmix-400b-shuffle. Users should review the upstream dataset card, composition, license information, and usage restrictions before using or redistributing this model.

Reproducibility notes

The default run configuration was read from the training code, while model dimensions and the final smoothed loss were verified against the saved 7,000-step checkpoint. The training plots above were generated from the accompanying CSV log. Hardware, exact compute dtype, random seed for model initialization, and independent evaluation results were not recorded in the checkpoint and are therefore not claimed here.

Citation

If you use this preview, cite the Hugging Face repository:

@misc{salem2026titanius1,
  author       = {Nadav Salem},
  title        = {Titanius-1-115m-base-fp16-preview},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/NadavSalem/Titanius-1-115m-base-fp16-preview}}
}