Veyra2-Mango-30M-Base

Veyra2-Mango-30M-Base

Veyra2-Mango-30M-Base is a 30.7M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation.

Model Details

Tokenizer

Special tokens:

• <|endoftext|>: 0
• <|im_start|>: 1
• <|im_end|>: 2
• <|pad|>: 3

Training Data

The model was trained on a 30B-token pretraining mixture.

Stage 1 18,000,000,000 tokens 180 shards

Mixture:

dclm_baseline: 50%
finephrase: 20%
cosmopedia_v2: 10%
finemath_4plus: 10%
ultrafineweb_multistyle: 5%
ultrafineweb_qa: 5%

Stage 1.5 4,000,000,000 tokens 40 shards

This stage linearly transitions from the Stage 1 mixture to the Stage 2 mixture.

Stage 2 8,000,000,000 tokens 80 shards

Mixture:

finephrase: 30%
dclm_baseline: 30%
cosmopedia_v2: 18%
finemath_4plus: 10%
ultrafineweb_multistyle: 5%
ultrafineweb_qa: 5%
ultrachat: 2%

Training Summary

• Final step: 25,432
• Tokens seen: 30,000,000,000
• Tokens per step: 1,179,648
• Sequence length: 3072
• Last train loss: 2.5062

Usage

<pre><code>import torch from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "veyra-ai/Veyra2-Mango-30M-Base" tokenizer = AutoTokenizer.from_pretrained( model_id, ) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.float16, device_map="auto", ) prompt = "In the 19th century" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=120, do_sample=True, temperature=0.6, top_p=0.9, repetition_penalty=1.1, use_cache=True, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True)) </code></pre>

Notes on Generation

Veyra2-Mango-30M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.

Intended Use

This model is intended for:

• small language model research
• continued pretraining
• benchmarking
• experimentation with compact causal LMs

Limitations

• Not instruction tuned
• Not RLHF tuned
• Not safe for factual or high-stakes use without additional validation
• Can hallucinate names, citations, species, references, and technical claims
• Open-ended text may drift off-topic
• Context length during training was 3072 tokens

Citation

If you use this model, please cite the model repository:

veyra-ai/Veyra2-Mango-30M-Base