dasha-215m

Dasha 215M (chat)

A decoder-only transformer trained from scratch (not fine-tuned from an existing pretrained LLM). This is the instruction/chat-tuned (SFT) version.

This is an early checkpoint: pretrained on a relatively small, general (Wikipedia ru+en) corpus. A larger, much more densely-trained successor (higher tokens/parameter ratio, broader web+math data, same architecture) is currently in progress -- this card will be updated when it replaces this checkpoint.

Architecture

• RMSNorm (pre-norm), RoPE, Grouped-Query Attention (GQA: 16 query heads / 4 KV heads), QK-Norm, SwiGLU MLP, tied embeddings
• 213.9M parameters, 16 layers, dim=1024, context length 2048
• torch SDPA attention with an automatic flash-attn fallback on supported architectures (Ampere/Ada/Hopper)

Training

• Optimizer: hybrid Muon (2D hidden-layer matrices) + Adam-mini (embeddings) + AdamW (norm gains)
• Pretrain data: Wikipedia ru+en
• SFT: ru/en multi-turn dialogue (Vikhrmodels/GrandMaster-PRO-MAX), ChatML-style format

Evaluation

Evaluated with lm-evaluation-harness (0-shot, acc) and the official MERA benchmark for Russian. Numbers are honest, not cherry-picked -- at 214M parameters and this data scale, the model is a fluent, grammatical talker with weak multi-step reasoning and weak math, which is the expected profile for this size class, not a bug.

English common-sense/reasoning (vs. Pythia-160M, a similarly-sized public baseline):

Russian (XWinograd-ru, XStoryCloze-ru, XNLI-ru, Belebele-ru):

MERA (aggregate, official submission): 0.195 -- near-random overall, with math/code tasks essentially at 0 (as expected: no dedicated math/code data in this checkpoint's pretrain mix). The in-progress successor targets this specifically with a denser, more diverse pretrain corpus.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("dariumi/dasha-215m")
model = AutoModelForCausalLM.from_pretrained("dariumi/dasha-215m", trust_remote_code=True, dtype=torch.bfloat16)

ids = tok("Привет!", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=50, do_sample=True, temperature=0.8)
print(tok.decode(out[0]))

Note: generate() via this HF wrapper reprocesses the full sequence each step (no KV-cache), so it is correct but not fast for long generations.