BananaMind-2-Medium

BananaMind-2-Medium

BananaMind-2-Medium is a decoder-only causal language model trained from scratch by BananaMind on a 50B-token curriculum.

The model has 49,559,552 parameters, a 3,072-token context window, and a custom 12,288-token digit-aware byte-level BPE tokenizer. It uses grouped-query attention, QK normalization, RoPE, SwiGLU, RMSNorm, tied input/output embeddings, and a KV cache for generation.

<img src="benchmarks.png" alt="BananaMind 2 Medium benchmark comparison" width="100%">

Model Details

Tokenizer

BananaMind-2-Medium uses a custom 12,288-token byte-level BPE tokenizer trained on 50 GiB of representative FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath-4+, and NPSet-2 Python educational data. It uses NFKC normalization and digit-aware pre-tokenization.

Digits are isolated before byte-level BPE, preventing the tokenizer from merging entire numbers into large number tokens.

Examples:

18  -> [20, 27]
227 -> [21, 21, 26]

Special token IDs:

Training Data

The 50B-token training mix combines educational web text, broad web text, synthetic textbook material, mathematics, and the complete local NPSet-2 Python educational corpus.

The run used a capacity-aware curriculum rather than sampling the aggregate mix uniformly from the first token.

Training Setup

Benchmarks

Self-reported scores using lm_eval and the official ArithMark 2.0 script. Scores may vary slightly by evaluation setup.

ARC Easy, ARC Challenge, PIQA, and HellaSwag use acc_norm,none. Independent Open SLM Leaderboard evaluation is not yet included.

Repository Files

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Medium"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Suggested Generation Settings

For stable continuations:

• do_sample=False
• repetition_penalty=1.1
• max_new_tokens=64 to 160

For more varied text:

• do_sample=True
• temperature=0.6 to 0.8
• top_p=0.9
• top_k=50
• repetition_penalty=1.1
• max_new_tokens=64 to 192

Intended Use

BananaMind-2-Medium is a base model intended for language-model research, local experimentation, text continuation, tokenizer research, arithmetic evaluation, and small-model training comparisons.

Because this is a base model, prompts should be written as continuation prompts rather than chat messages.

License

Apache 2.0

Benchmark Average Formula

The benchmark average groups both ARC tasks into one component:

ARC average = (ARC Easy + ARC Challenge) / 2
Average = (HellaSwag + ARC average + PIQA + ArithMark 2.0) / available component count

Missing components are omitted. If only one ARC score is available, that score is used as the ARC component. An average is reported only when at least two components are available.