typakos-140m-base

Typakos-140M-base

The name "Typakos" comes from the Greek "Τυπάκος," meaning "small dude," a nod to the model's small (140M) parameter count.

Typakos-140M-base is a 140M-parameter bilingual (Greek/English) base language model with a Llama-family architecture and a 2048-token context length. It was trained from scratch (no warm start), full-parameter, on approximately 9.33B tokens split roughly 50/50 between English and Greek. This is a base, completion-only checkpoint: it has no chat template and has not undergone any instruction tuning, so it should be prompted as a plain text continuation model, not as a chat assistant.

The full training pipeline, code, and configs live in scripts/typakos_140m/ on GitHub.

Model Details

Tokenizer

A byte-level BPE tokenizer with 50,000 learned merges plus 256 byte tokens and 2 special tokens (<|begin_of_text|>, id 0; <|end_of_text|>, id 1, which also doubles as the pad token). The pre-tokenizer regex pipeline is structurally borrowed from Llama 3.2, but the vocabulary itself was trained from scratch on the bilingual corpus below.

The tokenizer's post-processor wraps every encode in <|begin_of_text|> ... <|end_of_text|> by default. This is convenient for training but wrong for a generation prompt (a trailing EOS tells the model the prompt is already a finished document). See the usage example below for how to bypass it.

Training Data

Sourced with a 50/50 English/Greek token target:

Training Procedure

Reproduction

All code, Hydra configs, and step-by-step reproduction commands (data download, tokenizer training, shard packing, and the training launch script) are in scripts/typakos_140m/ in the pretrain GitHub repo, under the README.md there:

1. Environment setup and initial data pull (setup.sh)
2. Train the bilingual BPE tokenizer (train_tokenizer.py)
3. Measure bytes/token per source and derive the token mix plan (measure_token_rates.py)
4. Download the full corpus per that plan (download_data.py)
5. Mix sources into shuffled, proportional shards (concat_data.py)
6. Tokenize and pack into 2048-token sequences (prepare_shards.py)
7. Train (train.sh, pointed at configs/train.yaml)

How to Use

This is a base model, so prompt it with plain text and let it continue. Because the tokenizer auto-wraps every encode in BOS/EOS, use add_special_tokens=False and prepend BOS manually to avoid signaling a "finished document" to the model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "alexliap/typakos-140m-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "The history of Athens begins"
input_ids = tokenizer(
    tokenizer.bos_token + prompt, add_special_tokens=False, return_tensors="pt"
).input_ids

output = model.generate(input_ids, max_new_tokens=100, do_sample=True, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Evaluation

Evaluated 0-shot with EleutherAI lm-evaluation-harness, except GSM8K at 5-shot, matching the shot conventions stated on the SmolLM2 model card.

The gap to SmolLM2-135M-8k below is largely a training-budget gap, not an architecture gap: per its model card, SmolLM2-135M was pretrained on about 2T tokens, roughly 200 times the ~9.33B tokens Typakos-140M-base has seen, on a curated English-only mix (FineWeb-Edu, DCLM, The Stack, plus additional filtered sources). Typakos-140M-base is a single-stage, from-scratch run on a much smaller, bilingual (EN/EL) corpus, so it is comparable in size and architecture but not yet in data scale.

Greek Suite

Evaluated 0-shot on the ilsp_greek task suite (65 sub-tasks, including all 57 mmlu_greek subject splits). No SmolLM2 reference exists for these tasks, so they are reported standalone:

medical_mcqa_greek and mcqa_greek_asep have no English-benchmark equivalent and are reported for completeness only. winogrande_greek uses a different task formulation than the English winogrande row above (full-candidate-sentence plausibility scoring rather than fill-in-the-blank), so the two are not directly comparable.

Limitations

• It is a base, completion-only model with no instruction tuning or chat capability.
• It is best suited for continued pretraining, fine-tuning, or research experimentation, not for direct deployment as an assistant.

License

MIT