Downloads · 30 days
213
25% of all-time downloads
Aratako/ELYZA-japanese-Llama-2-MoE-2x13B-v0.1
ELYZA-japanese-Llama-2-MoE-2x13B-v0.1 is a text generation model from Aratako. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
Downloads · 30 days
213
25% of all-time downloads
All-time downloads
863
Public
Parameters
21.5B
43 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors43 GB · 100%
From the Hugging Face model README
Llama-2ベースの学習済み日本語モデルであるelyza/ELYZA-japanese-Llama-2-13bと、そのinstruction tuningモデルであるelyza/ELYZA-japanese-Llama-2-13b-instruct を、mergekitを使ってMoEを行い作成したモデルです。
以下2モデルを利用しています。
元モデルの通り、Llama2ライセンスを継承します。
Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
ベースとしたELYZA-japanese-Llama-2-13b-instructと本モデルのjapanese-mt-benchの結果は以下の通りです。 (シングルターン)
| Model | Size | Coding | Extraction | Humanities | Math | Reasoning | Roleplay | STEM | Writing | avg_score |
|---|---|---|---|---|---|---|---|---|---|---|
| ELYZA-japanese-Llama-2-13b-instruct | 13B | 3.7 | 6.0 | 6.6 | 2.4 | 2.5 | 5.2 | 5.8 | 7.2 | 4.925 |
| This model | 2x13B | 3.7 | 6.9 | 6.3 | 3.7 | 4.4 | 6.0 | 7.0 | 7.4 | 5.675 |

ベンチマークに使用したプロンプト
"""<s>[INST] <<SYS>>
あなたは誠実で優秀な日本人のアシスタントです。
<</SYS>>
{instruction} [/INST]"""
This model is created using MoE (Mixture of Experts) through mergekit based on elyza/ELYZA-japanese-Llama-2-13b and elyza/ELYZA-japanese-Llama-2-13b-instruct.
Click here for the GGUF version
It utilizes the following two models:
This model inherit the Llama2 license.
Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
The results of this model and the base ELYZA-japanese-Llama-2-13b-instruct on japanese-mt-bench are as follows. (Single turn)
| Model | Size | Coding | Extraction | Humanities | Math | Reasoning | Roleplay | STEM | Writing | avg_score |
|---|---|---|---|---|---|---|---|---|---|---|
| ELYZA-japanese-Llama-2-13b-instruct | 13B | 3.7 | 6.0 | 6.6 | 2.4 | 2.5 | 5.2 | 5.8 | 7.2 | 4.925 |
| This model | 2x13B | 3.7 | 6.9 | 6.3 | 3.7 | 4.4 | 6.0 | 7.0 | 7.4 | 5.675 |

Prompt used for benchmark
"""<s>[INST] <<SYS>>
あなたは誠実で優秀な日本人のアシスタントです。
<</SYS>>
{instruction} [/INST]"""
base_model: ./ELYZA-japanese-Llama-2-13b-instruct
gate_mode: random
dtype: bfloat16
experts:
- source_model: ./ELYZA-japanese-Llama-2-13b-instruct
positive_prompts: []
- source_model: ./ELYZA-japanese-Llama-2-13b
positive_prompts: []
tokenizer_source: model:./ELYZA-japanese-Llama-2-13b-instruct