BackKiyo-10M

BackKiyo-10M is a compact decoder-only language model from the Kiyo family, created for the backup account of DedeProGames.

The model was pretrained from scratch on 20 billion tokens drawn from FineWeb-Edu, DCLM-Baseline, and FinePhrase.

BackKiyo-10M uses a compact Qwen3-style decoder architecture with grouped query attention, RMSNorm, SwiGLU MLPs, RoPE positional embeddings, and tied input/output embeddings. The model contains just under 10 million parameters and was trained independently from a random initialization.

Model Details

Training Data

The complete pretraining run consumed approximately 20 billion tokens across this mixture.

Benchmarks

Self-reported results from the BananaMind Base Bench 1.1 suite.

The evaluation contains 350 examples, split across seven categories with 50 examples each. Each item contains four candidate continuations and is scored using conditional mean log-probability.

Overall Result

Detailed Results

* = passes 1.96σ vs. chance; n=50 per category.

By Difficulty

Benchmark Summary

The benchmark run reported 0 truncated sequences and 0 truncated continuations.

Scores are self-evaluated and may vary with benchmark revision, Transformers version, dtype, hardware, and evaluation settings.

Usage

Install the required dependencies:

pip install -U transformers safetensors torch

Load the model with Transformers:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DedeBckp/BackKiyo-10M"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=(
        torch.bfloat16
        if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
        else torch.float32
    ),
).eval()

if torch.cuda.is_available():
    model = model.cuda()

prompt = "The meaning of life is "

inputs = tokenizer(
    prompt,
    return_tensors="pt",
)

inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

BackKiyo-10M is a base language model, not an instruction-tuned model. It is designed primarily for text continuation rather than instruction following or conversational use.

At approximately 10 million parameters, the model has very limited capacity compared with larger language models. Benchmark results show its strongest performance in language completion, while world knowledge, reasoning, quantitative tasks, context tracking, and code completion remain substantially more limited.

The model may generate incorrect facts, inconsistent text, repetitive outputs, or lose coherence on difficult or long sequences. It should not be used for high-stakes decisions without independent verification.

License

Apache 2.0