RakutenAI-2.0-8x7B

---

license: apache-2.0

RakutenAI-2.0-8x7B

Model Description

RakutenAI-2.0-8x7B is an MoE-based foundation model derived from RakutenAI-7B, first introduced in March 2024. As part of a broader initiative to advance Japanese LLM technology, RakutenAI-2.0-8x7B adopts a Mixture of Experts (MoE) architecture with two active experts, resulting in 13B active parameters. This design enables dynamic expert selection based on input tokens, enhancing computational efficiency while maintaining high performance. RakutenAI-2.0-8x7B achieves state-of-the-art results on Japanese language understanding benchmarks while also demonstrating competitive performance on English evaluation tasks compared to similar models, including Swallow-MX-8x7B-NVE-0.1, Llama-3-Swallow-70B-v0.1, Sarashina2-70B, and PLaMo 100B.

If you are looking for an instruction-tuned model, check RakutenAI-2.0-8x7B-instruct.

Model Evaluation Results

<div style="text-align: center;">Table1: RakutenAI-2.0-8x7B foundation model average performance scores on LM-Harness in comparison with other Japanese open models.</div>

Detailed scores are as follows:

<div style="text-align: center;">Table2: RakutenAI-2.0-8x7B foundation model performance on LM-Harness metrics in comparison with other Japanese open models.</div>

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
model_path = "Rakuten/RakutenAI-2.0-8x7B"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="auto", device_map="auto")
model.eval()

requests = [
    "南硫黄島原生自然環境保全地域は、自然",
    "The capybara is a giant cavy rodent",
]

for req in requests:
    input_text = tokenizer(req, return_tensors="pt").to(device=model.device)
    tokens = model.generate(
        **input_text,
        max_new_tokens=512,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )
    out = tokenizer.decode(tokens[0], skip_special_tokens=True)
    print("INPUT:\n" + req)
    print("OUTPUT:\n" + out)

Note on Evaluation Scores:

• Evaluation tests were carried out on LM Evaluation Harness during October - December 2024. We use default task definitions from the following commit: https://github.com/EleutherAI/lm-evaluation-harness/commit/26f607f5432e1d09c55b25488c43523e7ecde657
• The tasks considered for Japanese evaluations are listed here: https://github.com/EleutherAI/lm-evaluation-harness/blob/26f607f5432e1d09c55b25488c43523e7ecde657/lm_eval/tasks/japanese_leaderboard/README.md
• The tasks considered for English evaluations are listed here: https://huggingface.co/docs/leaderboards/en/open_llm_leaderboard/archive https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/leaderboard/README.md

Model Details

• Developed by: Rakuten Group, Inc.
• Language(s): Japanese, English
• License: This model is licensed under Apache License, Version 2.0.
• Model Architecture: Mixture of Experts (2 active experts)

Limitations and Bias

The suite of RakutenAI-2.0 models is capable of generating human-like text on a wide range of topics. However, like all LLMs, they have limitations and can produce biased, inaccurate, or unsafe outputs. Please exercise caution and judgement while interacting with them.

Citation

For citing our work on the suite of RakutenAI-2.0 models, please use:

@misc{rakutengroup2025rakutenai2.0,
  author = {Rakuten Group, Inc.},
  title = {RakutenAI-2.0},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Rakuten},
}