JunkyAI

JunkyAI

A 15.7-million-parameter language model trained from scratch.

JunkyAI is a small English language model built end to end without any pretrained weights: its own byte-level BPE vocabulary, its own transformer implementation, its own training loop, and its own GGUF writer. It speaks simple, fluent English — roughly the register of a children's book — and answers short questions. It does nothing else, by design and by capacity.

It exists to be understood completely rather than to be capable. Every component is a few hundred lines of readable Python with no ML framework above PyTorch tensors: no transformers, no tokenizers, no peft, no gguf.

---

At a glance

The architecture is deliberately Llama-shaped so the exported GGUF loads in unmodified llama.cpp, Ollama and LM Studio. One detail differs from the usual HuggingFace implementation: RoPE is applied to interleaved pairs rather than a rotate-half split, matching ggml's GGML_ROPE_TYPE_NORM so weights export with no permutation step.

---

Training

Stage 1 — Pretraining

TinyStories is a corpus of short children's stories written with a small vocabulary. It was chosen because its distribution is close to what a model this size can actually represent — a general web corpus would exceed its capacity and produce worse English, not better.

Stage 2 — Instruction tuning

Only assistant tokens contribute to the loss; prompt tokens are masked out. Full fine-tuning was used rather than LoRA — at this size the model occupies under 4 GB of VRAM during training, so parameter-efficient methods save nothing and cost quality. LoRA and NF4 QLoRA are implemented in the repository for larger base models.

---

What it can do

• Hold a short, grammatical conversation in simple present and past tense
• Answer "What is X?" for everyday concrete nouns — animals, food, weather, household objects, places
• Greet, thank, say goodbye, and answer questions about itself
• Continue a short story coherently for several sentences
• Decline questions beyond its ability, with varied wording, and offer an alternative

Representative outputs:

How are you?               -> I am good, thank you.
hello                      -> Hi! I am happy to see you.
What is a dog?             -> A dog is a pet. It wags its tail and barks.
Tell me what rain is.      -> Rain is wet and cold.
What is 847 times 293?     -> That is too much for a small model like me.
                              But I can tell you about food or play.
Write me a Python program. -> I cannot work that out. I am sorry.
                              I can talk about simple things instead.

---

What it cannot do

These are capacity limits, not bugs, and no additional training fixes them at this parameter count.

• No arithmetic. It cannot add, multiply or count reliably.
• No code. It cannot read or write any programming language.
• No world knowledge. It knows nothing about real people, places, dates or events. Anything it says about them is invented.
• No reasoning. It cannot follow multi-step logic or answer "why" questions with genuine explanation.
• Long-range logic drifts. Within a few sentences it stays coherent; across a paragraph, causality breaks down. It may describe a character wanting something that belongs to a different character.
• Weak multi-turn. With only 179 conversations in training, it tends to lock onto the first topic and answer later, unrelated questions as if still on it. Single-turn use is markedly more reliable.
• Narrow vocabulary. Words outside the TinyStories range (~1,500–3,000 common English words) have near-random representations. It will produce them as noise.
• English only.

It will sometimes answer confidently and incorrectly. The graceful refusals cover common cases but are not a reliable safety property.

---

Intended use

Educational and experimental. JunkyAI is a complete, legible example of the full language-model pipeline at a scale that trains in an hour on one consumer GPU — useful for understanding tokenisation, transformer internals, pretraining, instruction tuning, quantisation and the GGUF format.

Not suitable for: production systems, factual reference, advice of any kind (medical, legal, financial), decisions affecting people, content moderation, or any setting where an incorrect answer carries cost. It has received no safety tuning or red-teaming beyond the refusal examples described above.

---

Data provenance and licensing

• Pretraining: TinyStories, by Eldan and Li (arXiv:2305.07759), distributed on the Hugging Face Hub as roneneldan/TinyStories under CDLA-Sharing-1.0. That is a share-alike data license: if you publish the data or data derived from it, you must do so under the same terms. Whether model weights count as derived data under CDLA is not settled law, so this card carries the same license as the safer assumption. TinyStories was itself generated by GPT-3.5 and GPT-4, so OpenAI's terms may also apply upstream.
• Instruction data: synthetic, generated through the Groq API using openai/gpt-oss-120b. Model outputs may be subject to the provider's terms; the data inherits whatever biases and errors that model has.
• Code: the training pipeline in this repository is separate from the weights and carries whatever license you choose for it. The license above applies to the released model, because of the corpus it was trained on.

The instruction data was generated and filtered automatically, and was not reviewed example by example.

---

Running it

llama-cli -m junkyai-q8_0.gguf -cnv

The chat template is embedded in the GGUF, so llama.cpp formats turns itself. Ollama, via a Modelfile:

FROM ./junkyai-q8_0.gguf
PARAMETER temperature 0.8
PARAMETER stop "<|eos|>"

Suggested sampling: temperature 0.7–0.8, top-k 40, top-p 0.95, repetition penalty 1.1, and 40–60 new tokens. Longer generations drift.

Ask one question at a time. Clearing the conversation between unrelated questions noticeably improves answers.

---

Reproducing

python scripts/bootstrap_pretrain_data.py     # ~10 min
python scripts/train_tokenizer.py             # ~20 s
python -m junkyai.train --stage pretrain --steps 9000
python scripts/generate_sft_data.py           # needs a Groq API key
python -m junkyai.train --stage sft --steps 300 --init-from checkpoints/junkyai-tiny
python -m junkyai.gguf_export --checkpoint checkpoints/junkyai-tiny-sft --quant q8_0

Verify the exported file against the checkpoint it came from:

python scripts/verify_gguf.py junkyai-q8_0.gguf --checkpoint checkpoints/junkyai-tiny-sft

---

Quantisation

Round-trip error measured against the source weights:

Q8_0 is the released format. Quantisation error is absorbed by redundant weights, and a 15.7M model has none to spare — Q4_0 saves 9 MB and does sixteen times the damage. Norm tensors are kept at F32 in all formats.