BananaMind-2-Nano

BananaMind-2-Nano

BananaMind-2-Nano is a compact decoder-only causal language model trained from scratch by BananaMind on a 30B-token curriculum.

The model has 9,968,128 parameters, a 4,096-token context window, and a custom 8k-token digit-aware byte-level BPE tokenizer.

Model Details

Credits to AxiomicLabs and GPT X2 125M for the architecture inspiration.

Tokenizer

BananaMind-2-Nano uses the same custom 8k byte-level BPE tokenizer as BananaMind-2-Mini. Digits are kept as separate tokens so numbers do not collapse into large number tokens.

Training Data

The run used a progressive curriculum, beginning web-heavy and gradually increasing synthetic textbook and mathematics data.

Training Setup

Evaluation

These are self-reported scores produced with lm_eval. Scores may vary slightly depending on the evaluation harness version, runtime settings, dtype, and environment.

All task scores use acc_norm,none. The average is the mean of ARC Easy, PIQA, ARC Challenge, and HellaSwag.

The unrounded average is 0.357659. Available unrounded task results are ARC Easy 0.361953, PIQA 0.559848, and ARC Challenge 0.233788.

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

pip install -U transformers safetensors torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Nano"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

License

Apache 2.0