dhamma-parrot-v01

dhamma-parrot-v01

A very small (5.6M parameter) GPT-style language model, trained from scratch on a narrow corpus of English-language Vipassana meditation texts that use Pali terminology. It is a hands-on research artifact from a "baby LLM" project, not a general-purpose assistant.

You can see it in action on Dhamma Parrot website.

What this is, and what it is not

This is a personal, non-commercial machine-learning experiment. The author's question was narrow and technical:

How much knowledge and structure can a small language model acquire if it is trained only on Dhamma/Vipassana texts, with no other text of any kind in the corpus?

No general web text, no books outside the domain, no code, no instruction data. Everything the model knows about English, about sentence structure, and about the subject matter comes from one small single-domain corpus. That constraint is the entire point of the experiment, and it explains most of the model's behaviour.

It is not:

• a source of Dhamma teaching, instruction, or guidance
• affiliated with, endorsed by, reviewed by, or connected to the Vipassana Research Institute (VRI), Dhamma.org, any Vipassana centre, or any teacher or assistant teacher
• a representation of what any teacher actually said
• a product, a service, or anything anyone should rely on

Its output is machine-generated text that will contain hallucinations, invented "quotations", doctrinal errors, and heavy repetition. See What the output looks like for measured numbers. If you want to learn about Vipassana, go to the original sources or to a course. Not here.

Read this first: it is small, and it emits tags

1. It really is tiny. 8 layers, hidden size 216, 3 attention heads, a 5000-token vocabulary and a 360-token context, trained on roughly 1.85M tokens of a single-domain corpus. For scale, that is about 5.6M parameters against the billions in models you may be used to. Expect short-range fluency, limited coherence, repetition, and a strong tendency to reproduce corpus phrasing. Judge it as a learning exercise.

2. It was trained on text marked up with structural tags, and it will generate them. The corpus is annotated with an XML-like grammar that encodes style and layout, and those 20 tags are real single tokens (ids 0-19) rather than text the tokenizer splits up. The model learned them as part of the language, so tags will appear in the output by design -- that is not corruption.

Every container tag has a matching closer (</topic:essay>, </list>, ...). <pad> and <unk> are ordinary special tokens.

You can use these to steer style: open with <topic:essay> for essay-like prose, <topic:recollection> for personal narrative, or <topic:qa> with <q>/<a> for a question-and-answer shape.

In the GGUF these tags are exported as USER_DEFINED rather than CONTROL, so llama.cpp renders them as visible text and matches them in prompts without needing --special. Marked CONTROL, they would silently render as zero bytes and you would see structure with no markup, which is confusing.

Usage

This is not a chat model. It is a pure text-continuation machine.

There was no instruction tuning, no chat fine-tuning, no RLHF and no alignment of any kind. The model has never seen a conversation, a system prompt, a role marker or a chat template. It does exactly one thing: given a piece of text, it predicts what text plausibly comes next, and then keeps going.

Practical consequences:

• Asking it a question does not get you an answer. It continues the text of your question -- often with more questions, or with prose that drifts around the topic.
• Instructions are not followed. "Summarize this", "answer briefly", "act as a teacher" are just words to continue, not commands.
• The <topic:qa> / <q> / <a> tags produce question-and-answer shaped text. That is imitation of the corpus layout, not instruction following, and the answers are not answers to your question in any reliable sense.
• It never stops on its own. There is no EOS token; you must bound generation with a token limit or a stop string.
• In chat UIs (LM Studio, Ollama, etc.) use raw completion mode. Chat mode wraps your input in template tokens the model has never seen, and output gets worse.

Write a prompt as the opening of a passage you want continued, not as a request.

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "theyur/dhamma-parrot-v01"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()

prompt = "<topic:essay>\nA good Vipassana meditator understands"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=120, do_sample=True,
                     temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=False))

Use skip_special_tokens=False if you want to see the tags.

llama.cpp / LM Studio / Ollama

The repo ships dhamma-parrot-v01.gguf (f16, no quantization -- at this size quantizing costs quality for no useful saving).

llama-cli -m dhamma-parrot-v01.gguf -c 360 --temp 0.8 -n 200 \
    -p "<topic:essay>\nA good Vipassana meditator understands"

In LM Studio, use raw completion mode, not chat. A chat template wraps the prompt in tokens this model has never seen. An ollama Modelfile is included.

Prompting notes

• Start plain prompts with a leading space. The tokenizer was trained with add_prefix_space=True; llama.cpp does not apply it on the BPE path, and encode(" " + s) == encode(s), so a literal leading space reproduces training-time tokenization. Prompts that open with a tag are unaffected.
• There is no EOS token. Nothing stops generation on its own -- bound it with max_tokens / -n, or stop on a closing tag such as </topic:essay>.
• Context is 360 tokens, shared between prompt and completion.

Model details

Training

Trained from scratch on ~1.85M tokens for 14000 steps (batch 96, lr 0.0002, dropout 0.125), with cross-entropy and label smoothing 0.1. Best validation loss 4.162 (perplexity 64.2) at step 14000. No instruction tuning, no RLHF, no alignment of any kind -- this is a base completion model.

Training data

51 English-language documents, roughly 7.6M characters / 1.85M tokens, all within the Dhamma/Vipassana domain. Sources:

• Material from the VRI website's section for old students. The author is an old student and accessed this material as such.
• Books purchased by the author, including freely-distributed VRI PDF editions.
• Publicly available articles and posts in which old students shared their impressions after courses. These are the basis for the <topic:recollection> style.
• One AI-generated research summary about the Vipassana tradition, written by a large language model -- 22,373 bytes / 22,035 characters / 5,657 tokens, i.e. 0.31% of the corpus -- included as general background prose. It is the only file in the corpus not written by a human, and some of the model's "encyclopedic" register comes from it rather than from tradition material.

What is deliberately NOT in the corpus

The model was not trained on course instructions or on evening discourses. No course audio, no discourse transcripts, no guided-meditation instructions, and no material intended to be received only within a course were used at any point.

The corpus is limited to material that was, in its original form, intended to be shared among old students or with the general public -- newsletters, journals, published books, letters printed in newsletters, biographical and explanatory articles, and old students' publicly shared course impressions.

The corpus itself is not distributed with this model, and no part of it is reproduced in this repository. Only trained weights are published here.

Preprocessing

The source texts were re-composed by hand: paragraph and line-break structure was rebuilt, and structural tags (<h>, <list>, <item>, <quote>, <topic:*>) were added to mark headings, lists, quotations and style. This changes layout and markup only -- the underlying wording is unchanged, and the tags are the reason the model emits them.

What the output looks like

Measured on ~1.0M generated tokens across the four sampling modes, so these are numbers rather than impressions.

Repetition and looping

This is the model's most visible failure mode, and it is worse than the memorization:

• trigram repetition ratio 0.11-0.16; unique-token ratio 0.45-0.49
• a single 12-token phrase repeated up to 47 times within one sample
• a single 5-gram repeated up to 599 times within one sample

Expect loops. A repetition penalty and a stop condition on a closing tag help a lot.

Hallucination

The model has no knowledge base, no retrieval and no grounding. It fluently produces statements that are wrong, including about Buddhist doctrine, history and practice, and it will invent quotations and attribute them to teachers. It has no notion of who said what. Nothing it outputs should be attributed to any teacher or treated as doctrinally correct.

Memorization

A model this small on a corpus this narrow can echo training text, so this was measured explicitly rather than assumed. Method: exact n-gram overlap between generations and the training corpus, with Pali encoding normalized on both sides so that Pali terms compare on their content and not on their byte encoding.

Share of generated n-grams that appear verbatim in the corpus:

Share of generated tokens lying inside any verbatim span of a given length (tags removed):

Reading: more than 98.5% of generated tokens are not part of any 12-token verbatim copy, and overlap decays geometrically as n grows -- the signature of recombination rather than recitation. The longest verbatim match found was 28 tokens including tags, 23 tokens of prose. Across ~1.0M generated tokens there were 128 spans of >= 15 tokens (87 distinct).

Inspecting those spans, most are formulaic material that repeats inside the corpus itself: course-eligibility criteria, metta well-wishing formulas, standard doctrinal enumerations, and canonical Pali phrases. A small number are single sentences traceable to one specific passage. Nothing approaching a paragraph, a section or a document is reproduced.

This is a property of the model, not a feature: do not treat output as original text, and do not use it to reproduce source material.

Provenance and validation

Exported from a PyTorch checkpoint via a HuggingFace GPTNeoXForCausalLM directory, with equivalence checked at every stage before the GGUF was written:

• exported model vs original checkpoint: worst absolute logit difference 8.23e-06, identical argmax at every position, identical greedy continuations
• tokenizer: 25 corpus lines re-encoded, token ids identical
• GGUF: all 100 tensors bit-identical to the checkpoint recomputed and cast to f16

config.json intentionally carries both rotary_pct and rope_parameters. Removing the legacy rotary_pct key silently breaks the model on transformers >= 5: partial_rotary_factor falls back to 0.25 and the rotary dimension becomes 18 instead of 72, with no error raised.

Limitations

• Repetition. The dominant failure mode; see the measured numbers above.
• Not factual. No knowledge base; states things that are wrong, including about Buddhist doctrine. Not a source of teaching or guidance.
• Memorization. Can reproduce short training phrases close to verbatim.
• Single narrow domain. Off-topic prompts produce meandering meditation prose.
• Vocabulary gaps. Only bytes seen in the corpus have tokens and there is no byte fallback, so unusual input (including a literal backslash, tab or CR) becomes <unk>.
• Context. Exact behaviour is only defined up to 360 tokens.
• No safety work of any kind. No alignment, no filtering, no refusal behaviour.
• Numerical note. llama.cpp's GELU is the tanh approximation while the PyTorch and transformers paths use exact erf, so GGUF output can diverge by the occasional token.

Intended use

Intended for research and education about small language models: studying what a model learns from a tiny single-domain corpus, tokenizer and tagging behaviour, memorization measurement, and training dynamics at small scale.

Not intended for any use as a source of information about Buddhism, Vipassana, or meditation practice, nor for any commercial use.

License, rights and contact

license: other, deliberately.

• The model weights, configuration, export tooling and this model card are released by the author for non-commercial research and educational use.
• The training corpus is third-party copyrighted material and is not distributed here. The author claims no rights in those source texts and grants none. Publishing these weights is not a license to any underlying work, and the author's hand-added markup and arrangement do not affect the rights of the original authors and publishers.
• No affiliation or endorsement. This work is independent of VRI, Dhamma.org, and any Vipassana centre, teacher or assistant teacher. Nothing here is authorized by them.

If you hold rights in any of the source material and object to this model being published, please open a discussion on this repository or contact the author, and it will be taken down.