Vertex-0.6-35M-Base

Vertex-0.6-35M-Base

A ≈34M-parameter base language model (Qwen3 architecture), pretrained from scratch on a single RTX 4060 Laptop GPU (8GB). Sibling of Vertex-0.6-100M, but fully independent: its own tokenizer, its own data pipeline, no shared weights or vocab.

Model

Tokenizer

Independent byte-level BPE (tokenizer.json), vocab 32000, trained on a ≈480K-document sample of the pretraining mix (≈85% web / ≈15% code). Special tokens: <|endoftext|> (bos/eos), <|pad|>. No chat/instruct special tokens — this is a raw base model.

Training data

≈1.50B unique tokens, trained for 2 epochs (≈3.0B tokens seen, ≈88 tokens/param):

• Web (≈1.286B tokens, 85.7%): openbmb/Ultra-FineWeb-L1 (English, Apache-2.0)
• Code (≈214M tokens, 14.3%): codeparrot/codeparrot-clean (deduplicated Python)

Documents were concatenated with <|endoftext|> separators and packed into fixed 1024-token blocks, shuffled independently each epoch.

Training recipe

• Single epoch-pair on one RTX 4060 Laptop 8GB, bf16 autocast + torch.compile
• AdamW (fused): lr 3e-3 peak, cosine decay to 10%, 300-step warmup, weight decay 0.1, betas (0.9, 0.95), grad clip 1.0
• Batch: 8 x 1024 tokens x 12 grad-accum = 98,304 tokens/step
• 30,517 optimizer steps total, ≈50.8K tok/s sustained (≈16h wall clock)
• Trained 2026-08-24 → 2026-08-25

Usage

from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-0.6-35M-Base")
tok = PreTrainedTokenizerFast(
    tokenizer_file=hf_hub_download("VertexResearch/Vertex-0.6-35M-Base", "tokenizer.json"),
    bos_token="<|endoftext|>", eos_token="<|endoftext|>", pad_token="<|pad|>",
)
out = model.generate(tok("The", return_tensors="pt").input_ids, max_new_tokens=50)
print(tok.decode(out[0]))

Notes

• Base model only: no instruction tuning, no chat format, no safety tuning. For the instruction-tuned variant, see VertexResearch/Vertex-0.6-35M-Instruct.
• English + Python centric; other languages are effectively untrained.

Limitations

These models are not the most coherent yet and need more tuning: expect rambling, repetition, and inconsistent answers, especially over longer generations.