TinyJLLM-Instruct

TinyJLLM-Instruct — instruction-tuned 100M model

Supervised fine-tuning of TinyJLLM on 83,145 curated instruction examples from six public datasets. Trained from scratch in the TinyLLM repository with response-masked loss and attention: response positions attend only to the prompt, which prevents the model from learning a copy shortcut.

Published recipe: 1 epoch over the curated subset (Dolly-15k, CodeAlpaca, oasst1, MetaMathQA capped at 8K), validation loss 4.71 (perplexity 111). A 2-epoch run reached 4.29 but memorized more — the 1-epoch model produced better answers in side-by-side evaluation.

Model family

Usage

The model uses a plain marker format (no chat template):

### Instruction:
What is the capital of France?

### Response:

Because of the response-masked training protocol, generation must use the same masking (response tokens attend only to the prompt). Plain causal model.generate() produces degenerate repetition. A self-contained transformers loop:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "jaweed123/TinyJLLM-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()

@torch.no_grad()
def respond(instruction, max_new_tokens=40):
    prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
    ids = tok(prompt)["input_ids"]
    L = len(ids)
    seq = ids + [tok.pad_token_id]          # dummy keeps response positions aligned
    for _ in range(max_new_tokens):
        inp = torch.tensor([seq])
        T = inp.shape[1]
        mask = torch.tril(torch.ones(1, 1, T, T, dtype=torch.bool))
        mask[:, :, L:, :] = False           # response rows ...
        mask[:, :, L:, :L] = True           # ... see the prompt only
        nxt = int(model(inp, attention_mask=mask).logits[0, -1].argmax())
        if nxt == tok.eos_token_id:
            break
        seq.append(nxt)
    return tok.decode(seq[L + 1:])

print(respond("What is the capital of France?"))

The project's own learnllm.inference.generate.generate(..., response_mask_start=...) implements the same protocol with sampling and a repetition penalty.

Training details

Two bugs worth knowing about are documented in SFT_TRAINING.md: a label off-by-one that made training look converged while generations were garbage, and the response-masking fix that removed the copy shortcut.

Limitations

At 100M parameters this model reproduces answer shapes more reliably than answer content: math and code are frequently wrong, multi-turn context is poorly tracked, and world knowledge is narrow. It is an educational artifact, not a production assistant.

Citation

@misc{tinyjllm,
  title  = {TinyJLLM: A 100M-Parameter Small Language Model Built From Scratch},
  author = {Jaweed, Abdul},
  year   = {2026},
  url    = {https://github.com/Abdul-Jaweed/TinyLLM}
}