Mossez-100M-Instruct

Mossez-100M-Instruct

Mossez-100M-Instruct is an experimental 100M-parameter Russian–English instruction-tuning research model derived from the final CPT-trained Mossez-100M-Base.

This is a compact research artifact. It is not a strong, reliable, safe, or production-ready assistant. Its purpose is to study the behavior and post-training limits of the Mossez-100M family; stronger practical behavior is expected from later, larger families.

Model details

Tokenizer v1.1 preserves every original ID from 0 through 31,999 and adds seven role/FIM tokens at IDs 32,000–32,006. Its tokenizer.json SHA-256 is e9551d84b9947f741763bf815a2d5f6bfcc47a3b67c73fcbf386223e8ed969be.

Intended use

The model is intended for compact-model research, local experiments, instruction-tuning analysis, reproducibility work, and testing inference pipelines. It is not intended for factual authority, autonomous action, safety-critical use, cybersecurity advice, or high-stakes decisions.

Usage

Install recent versions of torch, transformers, safetensors, and jinja2. The repository contains a clean FP32 master; Transformers can load it directly at FP16 runtime dtype on CUDA.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mossez-systems/Mossez-100M-Instruct"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype)
model.to(device).eval()

messages = [{"role": "user", "content": "Кратко объясни, что такое RoPE."}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=False,
        eos_token_id=[tokenizer.eos_token_id, 32003],
        pad_token_id=tokenizer.pad_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Keep the total prompt plus generated length within 1,024 tokens. The model may fail to emit <|end|> or EOS, so always set a finite max_new_tokens limit.

Training lineage

The selected weights follow this exact path:

1. final CPT-derived Mossez-100M-Base;
2. vocabulary resize from 32,000 to 32,007 with the seven new rows initialized to the mean of the original embedding rows, without changing old rows;
3. mixed assistant-only SFT v3 through checkpoint step 555;
4. concise assistant-only calibration v4 for 216 optimizer steps.

The selected lineage exposed the model to 9,973,153 non-padding tokens and 6,030,590 supervised assistant tokens after resize. v4 used FP16 autocast, fused AdamW, gradient checkpointing, micro-batch 8, accumulation 2, learning rate 1e-5, and seed 3407. System/user text, role prefixes, BOS, and padding were excluded from loss; assistant content, assistant termination, and final EOS were supervised.

OASST1 v1/v2 and first-turn v5 were completed comparison pilots but are not in the selected weight lineage. See TRAINING_REPORT.md and EVALUATION.md.

Evaluation and selection

Checkpoint selection combined teacher-forced validation/test loss with a fixed 31-case RU/EN diagnostic suite and manual review. v4 step 216 was selected by relative performance among the bounded v1–v5 experiments, not because it met a product-quality threshold.

The clean FP32 export was independently reloaded and verified with 100,098,048 parameters, tied embeddings, finite weights, and finite logits in a real CUDA forward pass. A separate FP16 validation export also passed, but is not duplicated in this repository.

Limitations

• Instruction following is weak in both Russian and English.
• Factual questions, arithmetic, JSON extraction, classification, translation, and exact formatting frequently fail.
• Responses may hallucinate, repeat, terminate early, or fail to terminate.
• Harmful-request refusal tests failed; the model must not be treated as a safety layer.
• The model may continue in an irrelevant style or produce superficially fluent but incorrect text.
• v3 included a smol-contraints subset later found to contain some prompt-to- answer leakage. It was removed from v4, but its influence may remain in the selected weights.
• Public conversational datasets may contain residual bias, errors, or undesirable material despite filtering.
• The context window is only 1,024 tokens and the model has no current-world knowledge guarantee.

Do not use this model for medical, legal, financial, security, or other high-stakes decisions.

Data and attribution

Selected-lineage SFT data was derived from pinned, filtered subsets of OpenAssistant OASST2 and HuggingFaceTB SmolTalk. The Base model retains its own FineWeb/FineWeb2/Wikipedia training history. Source datasets are not redistributed here. See DATASET_ATTRIBUTION.md and NOTICE.md for pinned revisions, licenses, and caveats.

License

Model weights and original repository materials are released under Apache-2.0. Third-party datasets and source content remain under their respective licenses and terms. See LICENSE and NOTICE.md.