NanoSakura-0.3B

Model Card for NanoSakura-0.3B (Ja -> Zh)

This model is a highly efficient, custom-built Hybrid Sequence-to-Sequence (Seq2Seq) model designed for Japanese to Chinese translation. It combines the deep understanding capabilities of a Transformer Encoder with the blazing-fast, memory-efficient generation of a Mamba2 (State Space Model) Decoder.

Model Details

Model Description

Traditional Seq2Seq models (like T5 or BART) rely entirely on Transformers. While powerful, the self-attention mechanism in the decoder leads to an $O(N^2)$ computational bottleneck and high KV-cache memory usage during text generation.

This model solves that by introducing a Hybrid Architecture:

1. Encoder (Transformer): Uses Self-Attention + RoPE + SwiGLU to fully capture the global context of the source Japanese text in parallel.
2. Decoder (Mamba2 + Cross-Attention): Replaces self-attention with Mamba2's State Space Model (SSM). This allows for $O(1)$ state-updating generation (no growing KV cache), while retaining Cross-Attention to accurately "look back" at the encoder's features to prevent hallucination.

With only ~287 Million parameters, it achieves excellent translation quality while maintaining an extremely low hardware footprint, making it ideal for edge deployment or high-throughput API services.

• Developed by: telecomadm1145
• Model type: Hybrid Transformer-Mamba2 Seq2Seq
• Language(s) (NLP): Japanese (ja), Chinese (zh)
• License: MIT
• Parameters: ~287M

How to Get Started with the Model

Because this model uses a custom architecture, you must use trust_remote_code=True when loading it with the transformers library. The custom modeling_mamba2_s2s.py will handle the $O(1)$ Mamba2 cache generation automatically.

Use the code below to get started:

import torch
from transformers import AutoModelForSeq2SeqLM, PreTrainedTokenizerFast

repo_id = "telecomadm1145/NanoSakura-0.3B"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(
    repo_id, 
    trust_remote_code=True, 
    dtype=torch.float32 
)
model.to(device)

text = "おはようございます、今日の天気はいいですね！"
input_ids = tokenizer.encode(text + "<eos>")
input_tensor = torch.tensor([input_ids]).to(device)

output_ids = model.generate(
    input_tensor, 
    max_new_tokens=256,
    bos_token_id=1, 
    eos_token_id=2
)
result = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(f"Translation: {result}")
# Output: 早安，今天的天气真好呢！

Evaluation

We evaluated the model on the FLORES-200 benchmark (Japanese to Chinese, ja-zh) and a testset(shard_00134). To provide a comprehensive assessment, we report both lexical-overlap metrics (SacreBLEU) and semantic-similarity neural metrics (COMET).

Metrics

All evaluated using greedy decoding.