tiny-qwen3-30m

Tiny Qwen3 30M

Tiny Qwen3 30M is a 29.57M-parameter English causal language model trained from scratch on the TinyStories dataset. It is intended as a compact educational model for studying tokenizer training, pretraining, checkpointing, and text generation with the Hugging Face ecosystem.

This is a base language model, not an instruction-tuned or chat model.

Dataset

The model uses roneneldan/TinyStories, a synthetic English dataset of short stories generated to contain vocabulary and concepts that are typically understood by young children. Both the upstream train and validation splits are read in streaming mode.

Data preparation:

• The training stream is shuffled with seed 42 and a 10,000-example shuffle buffer.
• Empty stories are discarded.
• Stories are tokenized without automatically added special tokens, then separated with <|endoftext|>.
• Tokens from consecutive stories are concatenated and packed into fixed 512-token sequences without padding.
• Validation uses the same packing procedure and is limited to 256 packed sequences (131,072 tokens) per evaluation.

Tokenizer

A byte-level BPE tokenizer is trained from scratch on 100,000 TinyStories training examples. It uses NFC normalization, an 8,192-token vocabulary, and the special tokens <|endoftext|>, <unk>, <|im_start|>, and <|im_end|>. The tokenizer supports sequences up to 1,024 tokens, while training uses 512-token sequences.

Model architecture

Training method

The model is pretrained from random initialization with the standard next-token causal language-modeling objective. Training uses the Hugging Face Trainer, streaming packed data, mixed precision when supported, gradient accumulation, periodic validation, and checkpoint-based resume.

Training parameters

The recorded run completed 20,000 optimizer steps on an NVIDIA GeForce RTX 4070, with a final aggregate training loss of approximately 0.1317. This value is reported for reproducibility and should not be treated as a benchmark against other models.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "richarddzh/tiny-qwen3-30m"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "Once upon a time,"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Reproducibility

The complete training and upload workflow is available in the GitHub notebook.

Limitations

This small experimental model is trained only on synthetic English children's stories. It has limited factual knowledge, reasoning ability, context handling, and multilingual capability. Generated text may be repetitive, inconsistent, factually incorrect, or reflect artifacts in the source dataset. Do not use it for high-stakes decisions or assume its outputs are accurate.