TinkaaLM

TinkaaLM

A 1.94M-parameter language model, trained from scratch on one laptop GPU.

Code, configs, training scripts and the full write-up: https://github.com/WinterBlossom0/TinkaaLM

It writes grammatical English, follows the shape of an instruction, and knows almost no facts. Every number on this page was measured on this model.

Q: Which season is cold, winter or summer?
A: the winter season is cold, and winter is cool, but of cold, winter.

Q: Give me three tips for sleeping better.
A: For sleeping well, sleeping well. Researching the front of a sleeping base
   is crucial. Here are three tips:

   First, use the skin to get to eat and maximize your sleeping habitat.

It picks winter over summer. It produces Here are three tips: followed by First, for a question that asked for three. The content after that is nonsense. That gap — form without content — is the whole story.

Loading it

This model uses a custom architecture, and modeling_tinkaalm.py re-exports the real classes from the tinkaalm package rather than copying them — so an export can never drift from the code that trained it. That package must be installed or from_pretrained will fail:

git clone https://github.com/WinterBlossom0/TinkaaLM.git
cd TinkaaLM && pip install -e .

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("WinterBlossom/TinkaaLM", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "WinterBlossom/TinkaaLM", trust_remote_code=True).cuda().eval()

# chat format: <|user|> your text <|assistant|> ... <|endoftext|>
ids = torch.tensor([[tok.convert_tokens_to_ids("<|user|>"),
                     *tok("What colour is the sky?", add_special_tokens=False).input_ids,
                     tok.convert_tokens_to_ids("<|assistant|>")]]).cuda()
out = []
with torch.no_grad():
    for _ in range(40):
        # the head was trained with its last 3 rows cut out of the loss
        logits = model(ids).logits[0, -1, :4096].float()
        probs = torch.softmax(logits / 0.8, -1)
        order = torch.argsort(probs, descending=True)
        ranked = probs[order]
        keep = (torch.cumsum(ranked, 0) - ranked) < 0.95     # nucleus, p=0.95
        nxt = order[keep][torch.multinomial(ranked[keep], 1)]
        if nxt.item() == tok.convert_tokens_to_ids("<|endoftext|>"):
            break
        out.append(nxt.item())
        ids = torch.cat([ids, nxt.view(1, 1)], 1)
print(tok.decode(out))

Or just use the repo's own chat loop, which handles history and truncation:

python scripts/ask_chat.py

The reported outputs use nucleus sampling at T=0.8, p=0.95. Greedy decoding also runs and sometimes reads better, but it shows one mode rather than what the model actually puts probability on — at this size that difference matters.

The model

Three KDA layers to one attention layer is the 3:1 ratio from Kimi Linear. KDA is recurrent, so it carries position by itself and the attention layer needs no rotary embeddings. The MLPs use xIELU.

Head rows 4096–4098 (the role tokens) were never scored during training. They hold 0.005% of the softmax mass on held-out text, so leaving them in is harmless — but cutting logits to [:4096] is exactly what the model was trained with.

How it was trained

Each stage's loss is over its own validation mixture, so the three numbers measure three different things and do not form a trend.

Stages 2 and 3 rank neurons by exact ablation — zero one neuron, re-measure the loss, keep the difference (the zero-patch indirect effect of Marks et al. 2024) — then hold the most important ones bit-exactly while training everything else.

It did not work. Same weights, same evaluation — the whole BabyLM dev set, 19,675,583 tokens — before and after 4 epochs of Natural Questions with the top 16 neurons per layer (16,384 weights, 0.84% of the model) held bit-exactly:

99.16% of the weights were still training and the model lost 1.97 nats. Holding a small set of important weights still is not sufficient to prevent forgetting at this scale. No run was ever executed that holds task data, epoch count and frozen-neuron count fixed while varying the data mixture, so nothing here establishes what a mixture does.

Benchmarks

Zero-shot, options scored by log likelihood (lm-evaluation-harness). z is standard errors from chance — under 2 means guessing.

Grammar is the one thing it genuinely learned. BLiMP is 10.5 points over chance at z = 62 across 67,000 minimal pairs. It knows which of two sentences is well-formed, which is what a corpus of child-directed speech should teach.

SciQ supplies a support paragraph containing the answer. Its 64% measures picking an answer out of text placed in front of the model, not recall.

Four tasks land below chance, and two measured artifacts explain it. BoolQ's 37.83% is exactly the share of no answers in its validation set (1,237 of 3,270) — the model answers no to every question, a constant predictor rather than a poor guesser. OpenBookQA and ARC-Challenge are length effects: normalising by option length moves OpenBookQA from 11.20% to 26.60% and ARC-Challenge from 17.06% to 22.27%, both back to roughly chance. Unnormalised likelihood favours the shorter option, which on those two tasks is reliably wrong.

Limitations

It is a 1.94M-parameter research model. It has no factual knowledge, no reasoning, no safety training and no alignment. It will produce confident nonsense on any question of fact. Do not deploy it for anything.

License

MIT.