BananaMind-2-Pro-Preview-Chat

BananaMind-2-Pro-Preview-Chat

BananaMind-2-Pro-Preview-Chat is the instruction-tuned version of BananaMind-2-Pro-Preview. It was fully fine-tuned for one epoch on HuggingFaceTB/smol-smoltalk, with loss applied only to assistant content and the assistant-ending EOS token.

The model has 138,971,520 parameters, a 3,072-token context window, and a custom 32,768-token digit-aware byte-level BPE tokenizer. It supports system prompts, multi-turn conversations, grouped-query attention, QK normalization, RoPE, tied embeddings, and KV-cached generation.

This chat model is based on the 96K Pro Preview checkpoint, which was pretrained on 51,904,512,000 tokens. It is a preview-derived instruction model rather than a fine-tune of the completed 100B-token BananaMind 2 Pro run.

Model Details

Benchmarks

BananaMind Instruct Bench 1.1

Self-reported results from the official BananaMind Instruct Bench 1.1 script. The benchmark contains 300 deterministic, difficulty- and category-weighted instruction tasks. The BananaMind models used their native chat templates; Supra 1.5 used the benchmark's Alpaca-style fallback. Evaluation used greedy decoding, repetition_penalty=1.1, and seed 42.

Detailed final result

The complete final run used CUDA, bfloat16, the native chat template, and no sampling. Scores can vary with benchmark revision, Transformers version, dtype, hardware, and generation settings.

Fine-tuning Progression

The final export is the strongest measured checkpoint overall. Step 3,000 had a slightly higher code-category Elo of 1,345, but the final model passed more code tasks and scored higher overall.

Instruction Tuning

System and user messages were retained as context but masked from the loss. Only assistant content and the assistant-ending EOS token contributed to the objective. All model parameters were trainable; no adapters or LoRA modules were used.

Chat Template

The tokenizer includes a Jinja template for system, user, and assistant messages:

<BOS><|system|>
{system message}
<|user|>
{user message}
<|assistant|>
{assistant response}<EOS>

The role markers are plain text encoded by the existing tokenizer. Fine-tuning did not add or resize any tokens, and the input and output embeddings remain tied.

Usage

This repository contains custom Transformers architecture code and must be loaded with trust_remote_code=True.

pip install -U torch transformers safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Pro-Preview-Chat"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if device == "cuda" and torch.cuda.is_bf16_supported()
    else torch.float32
)

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

messages = [
    {
        "role": "system",
        "content": "You are a concise and helpful assistant.",
    },
    {
        "role": "user",
        "content": "Write a Python function that squares a number.",
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        repetition_penalty=1.1,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
        use_cache=True,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

For multi-turn chat, append the generated assistant response and the next user message to messages, then render the full conversation with the chat template again.

Suggested Generation Settings

For stable responses:

• do_sample=False
• repetition_penalty=1.1
• max_new_tokens=128 to 256
• use_cache=True

For more varied responses:

• do_sample=True
• temperature=0.6 to 0.8
• top_p=0.9
• repetition_penalty=1.1
• max_new_tokens=128 to 256

Keep a finite generation limit. The combined prompt and generated output must fit within the 3,072-token context window.

Repository Files

Intended Use and Limitations

BananaMind-2-Pro-Preview-Chat is intended for small-model research, local chat experiments, educational demonstrations, code-generation experiments, instruction-tuning studies, and compact-model comparisons.

The model has not received dedicated safety alignment. It can hallucinate facts, fail arithmetic or logical tasks, produce insecure or invalid code, misunderstand instructions, and generate biased, repetitive, or otherwise undesirable text. Do not rely on it for medical, legal, financial, safety-critical, or other high-stakes decisions.

License

This repository is released under the BananaMind Community License 1.0. Commercial products or services exceeding either threshold in Section 1 require a separate commercial license from Banaxi-Tech.