Vertex-0.6-100M-8192-ctx-Base

Vertex 0.6 100M — 8192-ctx Base

Base (text-completion) model of the Vertex 0.6 family by Vertex Research, pretrained from scratch on 10.01B tokens at 1024 context, then context-extended to 8192 via a three-stage rope ladder (2048: 350M tokens; 4096, theta 250K: 300M tokens; 8192, theta 1M: 285M tokens) on fresh Ultra-FineWeb. This is the raw pretrain checkpoint: no instruction tuning, no chat alignment. Instruct and reasoning variants (with extended context) are released separately.

Model details

Training

• Data: ~10.01B tokens — English web text from Ultra-FineWeb plus a code slice, packed into 1024-token blocks, seeded shuffle, single epoch.
• Schedule: 101,819 optimizer steps, global batch 98,304 tokens (12 × grad-accum 8 × 1024), LR 3e-3 cosine → 3e-4, 300 warmup steps.
• Hardware: single RTX 4060 Ti 16GB, ~35K tokens/s, ~3.5 days wall clock.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("VertexResearch/Vertex-0.6-100M-8192-ctx-Base")
model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-0.6-100M-8192-ctx-Base")

ids = tok("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=30)
print(tok.decode(out[0]))

This is a completion model — prompt it with text to continue, not with chat messages.

Limitations

These models are not the most coherent yet and need more tuning: expect rambling, repetition, and inconsistent answers, especially over longer generations.

97M parameters and 10B tokens: expect fluent English and basic factual/code completion, frequent factual errors, and repetition under greedy decoding. Knowledge cutoff of the web corpus is approximately April 2024.