saslm-18m-mula

SASLM-18M (primary text only)

An 18.7M-parameter GPT-2-style language model trained from scratch on the Complete Works of Sri Aurobindo. Released as the reference model for the paper Commentary as a Substitute for Pretraining.

This model is not competitive with general-purpose language models and is not intended to be. It exists to support a research question about training data composition. See What this model is for and Limitations before using it for anything.

What this model is for

The paper asks whether adding secondary material — recorded commentary on a primary corpus — improves a small model on that primary corpus. This checkpoint is the control arm: trained on primary text only, no commentary. It is released so that others can reproduce the comparison.

The commentary-trained arms are not released pending permission from the commentary's author. See Training data.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("maheshcr/saslm-18m-mula")
tok = AutoTokenizer.from_pretrained("maheshcr/saslm-18m-mula")

ids = tok("The Supermind is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True,
                     temperature=0.8, top_k=50, pad_token_id=0)
print(tok.decode(out[0]))

Context length is 256 tokens. The tokenizer is a 30,000-token BPE trained on the domain corpus, so it is efficient on this vocabulary and poor on general text.

Model details

The released weights are bit-identical to the checkpoint used in the paper; conversion to the transformers layout was verified to zero logit difference.

Evaluation

Held-out validation set of 337,514 tokens (1,502,530 bytes), split at the document level and decontaminated to 0.3% verbatim overlap with training.

Bits per byte is used for cross-model comparison because the tokenizers differ; per-token perplexity is not comparable across different vocabularies.

Note the last row: a fine-tuned GPT-2-small is substantially better than this model. If you want a good model for this corpus, fine-tune a pretrained one. This model beats GPT-2-small only in the zero-shot setting, at 6.6x fewer parameters.

On SASLM-Eval (see the dataset card), this model scores 36.0% / 29.1% / 35.5% across three seeds on a 406-item cloze task with a 25% chance baseline. Above chance, but no model we tested is distinguishable from another on this benchmark.

Training data

This checkpoint was trained on Sri Aurobindo's works only. Sri Aurobindo died in 1950, so the works are out of copyright in India under life+60; the 2005 edition used carries a Sri Aurobindo Ashram Trust copyright notice. The corpus itself is not redistributed. A manifest with SHA-256 hashes is in the code repository so that independently obtained sources can be verified as identical.

Memorisation

Measured before release. 100 greedy prompts of 50 tokens per split, generating 100 tokens and measuring the longest matching continuation:

The model does not reproduce its training data. The longest verbatim match found anywhere was six tokens.

Limitations

• Small and undertrained. Validation loss was still improving at the step budget; this is not a converged model.
• Worse than fine-tuning a pretrained model on the same data, by 13.7% bits per byte. Do not use this model where quality matters.
• 256-token context. It cannot handle long passages.
• Single domain, single author, single register. Generations imitate the surface style of early-20th-century English philosophical prose and will be incoherent outside it.
• It does not know what it is saying. On a benchmark testing whether the model selects the correct ontological term from a same-family set, it scores around 33% against a 25% baseline. Fluent output should not be read as doctrinal accuracy.
• Not suitable for religious, spiritual or scholarly guidance. It generates plausible-sounding text in the register of a spiritual teacher without any correspondence to what that teacher actually taught. Do not present its output as Sri Aurobindo's views.

Citation

@misc{saslm2026,
  title  = {Commentary as a Substitute for Pretraining: Mixing Secondary Text
            into a Small Language Model for a Philosophical Corpus},
  author = {Mahesh C.R.},
  year   = {2026}
}

Contact

If you are a rights holder with a concern about this release, open an issue on the model repository and it will be addressed.