Vantora-Micro

Vantora Micro

A 9,800-parameter pure Llama-style causal language model, trained on a 100M-token slice of FineWeb-Edu. This is the "pure transformer" baseline in a head-to-head comparison against a hybrid Mamba-2 + attention model of the same size.

Model Details

Training

• Data: first 100M tokens of HuggingFaceFW/fineweb-edu sample-10BT
• Epochs: 1 (100M total tokens seen)
• Batch: 128 × seq 256 (3,051 steps)
• Optimizer: AdamW, lr 5e-3, cosine schedule + 15% warmup
• Grad clip: 1.0, seed 42
• Hardware: NVIDIA GTX 750 (Maxwell, 4 GB VRAM)
• Time: ~4.3 minutes

Benchmark: BananaMind Base Bench 1.1

Evaluated with the official BananaMind benchmark.py runner (official_complete_run: true, exact 350-item split, SHA-256 verified).

⚠️ Length-bias caveat on Code Completion

The Code Completion score (Elo 982, 34%) is not evidence the model can code. It is a benchmark artifact:

• In the code_completion category, the correct answer is the longest continuation 68% of the time (vs 12-34% in every other category).
• This model has a length bias: it picks the longest continuation more often than random, because its token distribution is near uniform and longer sequences accumulate more probability.
• The two effects line up, so the length bias coincidentally matches the correct answer most of the time.

The BananaMind README itself warns: "Mean token log-probability reduces direct continuation-length bias but does not eliminate every tokenizer-dependent effect." Treat the Code Completion Elo as a length-bias artifact, not a real coding skill.

vs. Vantora-Micro-Hybrid (same size/data)

The hybrid edges out this model by +53 Elo and +4.3% accuracy, but most of that gap comes from the length-bias artifact on Code Completion, not real reasoning. On PIQA / HellaSwag / ARC-Easy the two are within noise (0.5-2%).

Why this model is the practical choice

For a 10K-param model on a 100M-token slice of web text, the pure transformer is the better tradeoff:

• 11.5× faster to train (4.3 min vs 49.5 min on the same GTX 750).
• Within noise of the hybrid on every benchmark that measures real ability.
• No custom architecture, no trust_remote_code, loads with stock AutoModelForCausalLM.

The hybrid's SSM sequence memory was a clear win on TinyStories (where narrative memory mattered), but on this benchmark the extra training time buys almost nothing. This model gets the same result in a fraction of the time.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("VantoraLabs/Vantora-Micro")
tokenizer = AutoTokenizer.from_pretrained("VantoraLabs/Vantora-Micro")

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Files

config.json              # LlamaForCausalLM config
model.safetensors        # 9,800-param weights
tokenizer.json          # ByteLevel BPE (1024 vocab)
tokenizer_config.json   # tokenizer settings
special_tokens_map.json # special token mapping
generation_config.json  # generation defaults

Notes

This is an extremely small model — it is a research artifact for studying scaling laws and architecture comparisons at the sub-10K parameter scale, not a production language model. Its BananaMind score (Elo 810) is near the four-choice random baseline (25%), as expected for a model this size.