os24-125m-base

os24-125m-base

125.08M-parameter decoder-only English causal language model pretrained from scratch on a deterministic OpenSubtitles 2024 English corpus.

Model summary

os24-125m-base is a base causal language model, not an instruction-tuned assistant. It was trained from scratch on a deterministic packed English subtitle corpus derived from OpenSubtitles 2024.

The release preserves the final trained Safetensors weights byte-for-byte and adds a custom Hugging Face transformers integration. Because the architecture is custom, loading requires explicit trust_remote_code=True.

Architecture

Training data and budget

The production corpus is a deterministic OpenSubtitles 2024 English repack.

• source training tokens: 3,126,945,600
• source-token budget: 25.0 tokens / parameter
• final optimized prediction cursor: 3,126,943,744
• accepted untrained tail: 1,855 prediction tokens
• raw training data is not bundled with this model

Context curriculum: 512 (35%), 1,024 (30%), 2,048 (25%), 4,096 (10%).

The optimizer was AdamW with global-token linear warmup followed by cosine decay; the selected peak learning rate was 3e-4.

Final sealed held-out evaluation

Perplexity measures next-token modeling quality on this corpus distribution. It is not an instruction-following, reasoning, factuality, coding, or safety benchmark.

Intended use

Useful candidate applications include English dialogue/subtitle-style continuation, creative dialogue and scene continuation, small-base-model research, and downstream fine-tuning for summarization, classification, emotion/context, and instruction following.

Limitations and risks

• This is a base LM, not a chat/instruction model.
• Its training distribution is heavily dialogue/subtitle oriented.
• It can reproduce profanity, stereotypes, unsafe language, or other material occurring in film/TV subtitle text.
• It is not a factual authority and has no dedicated safety-alignment stage.
• The release wrapper currently performs generation without KV-cache reuse. This is mathematically conservative but slower than cache-optimized generation.
• Hosted inference support for custom code can differ by provider; local Transformers loading is the release's verified compatibility target.

Transformers loading

Review the repository code before opting in to custom code:

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "everarpitraj/os24-125m-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("I never thought I would see you again.", return_tensors="pt")
out = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    use_cache=False,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

For offline use, unzip the release ZIP and replace repo with the extracted directory.

Reproducibility / provenance

• training revision: OS24-125M-REAL-PRETRAIN-R3-CONTROL-HARDENED
• release revision: OS24-125M-BASE-HF-RELEASE-V1
• released weight SHA-256: 04b22149cac3297f33aa9c1af6308b4d164ce345fd8ecd9392d98d106a456c4f
• tokenizer SHA-256: a8ea0fd42e69835d86b3606545ae47a7edbc982a0cc153ab7806cb9266427ea4
• dataset manifest SHA-256: 7ef1ab2f410ffc983ad33561a01cd199ac54457ae89130c166450e6beb8c4f9b
• training plan SHA-256: 42d59bf8b53ffeb4a86a117b5028fad8676b2b81ffeea7b82cd566c5a26c4f14
• source bundle SHA-256: e467f45a1e81a9deef5e3c2292b12e513f3c75edca28a21f28af2c83dafe1341
• final checkpoint-state SHA-256: a3ebf2fb9e77293bb82235237d5d96d501a5c4d040e3c6bac5f21cd7a03a12ed

See PROVENANCE.json, FINAL_EVAL.json, MANIFEST.json, and LOCAL_VALIDATION.json.

License and source-data terms

Model license: not specified in this release configuration. Choose an appropriate license before making the repository public.

The training dataset itself is not redistributed in this repository. OpenSubtitles and underlying subtitle material have their own applicable terms and rights; the publisher and downstream users are responsible for evaluating those terms for their intended use.

Citation

No paper/DOI is invented by this release. Add a formal citation here if a technical report or paper is published later.