elroy-350m-base

elroy-350m-base

elroy-350m-base is a 361M-parameter Llama-style decoder trained from scratch on about 20B tokens of educational Python and English by Strife Technologies, as a fully documented teaching project. Every line of the tokenizer, model, training loop and fine-tuning code is in the repository, with a chapter explaining it:

https://github.com/strifero/elroy

This is the base variant: a text continuer. Prompt it with the start of a Python file (a signature and docstring, a comment, an import). It also completes gaps with <|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>.

Use

No transformers needed. elroy_min.py in this repo holds the model, the tokenizer and a sampler in one file:

import torch
from elroy_min import load, generate

model, tok = load(".")          # this folder
ids = tok.encode("def is_prime(n):\n    \"\"\"Return True if n is prime.\"\"\"\n")
out = generate(model, ids, max_new_tokens=128, temperature=0.2, top_p=0.95, stop_ids={tok.eot})
print(tok.decode(out))

Training

Evaluation

Data and licenses

Stack-Edu (HuggingFaceTB, Python subset; files with a permissive or no detected license, as in The Stack v2), StarCoderData (bigcode, permissive), FineWeb-Edu (ODC-By), Cosmopedia-v2 (ODC-By). Fine-tuning data: Magicoder OSS-Instruct (MIT), evol-codealpaca-v1 (Apache-2.0), and a handwritten set in the repository.

Limitations

This is a small model. It writes short, mostly correct Python functions and explains beginner concepts. It makes confident mistakes, knows little outside Python, has no knowledge of recent libraries, and has not been trained to refuse anything. Run what it writes before you rely on it.