Downloads · 30 days
377
36% of all-time downloads
Aratako/ELYZA-japanese-Llama-2-fast-MoE-2x7B-v0.1
ELYZA-japanese-Llama-2-fast-MoE-2x7B-v0.1 is a text generation model from Aratako. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
Downloads · 30 days
377
36% of all-time downloads
All-time downloads
1.1K
Public
Parameters
11.2B
22.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors22.3 GB · 100%
From the Hugging Face model README
Llama-2ベースの学習済み日本語モデルであるelyza/ELYZA-japanese-Llama-2-7b-fastと、そのinstruction tuningモデルであるelyza/ELYZA-japanese-Llama-2-7b-fast-instruct を、mergekitを使ってMoEを行い作成したモデルです。
以下2モデルを利用しています。
元モデルの通り、Llama2ライセンスを継承します。
Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
ベースとしたELYZA-japanese-Llama-2-7b-fast-instructと本モデルのjapanese-mt-benchの結果は以下の通りです。 (シングルターン)
| Model | Size | Coding | Extraction | Humanities | Math | Reasoning | Roleplay | STEM | Writing | avg_score |
|---|---|---|---|---|---|---|---|---|---|---|
| ELYZA-japanese-Llama-2-7b-fast-instruct | 7B | 2.8 | 5.2 | 7.1 | 2.0 | 3.6 | 6.0 | 5.9 | 6.4 | 4.8750 |
| This model | 2x7B | 3.5 | 5.1 | 7.5 | 1.9 | 3.5 | 6.3 | 5.9 | 7.6 | 5.1625 |

ベンチマークに使用したプロンプト
"""<s>[INST] <<SYS>>
あなたは誠実で優秀な日本人のアシスタントです。
<</SYS>>
{instruction} [/INST]"""
This model is created using MoE (Mixture of Experts) through mergekit based on elyza/ELYZA-japanese-Llama-2-7b-fast and elyza/ELYZA-japanese-Llama-2-7b-fast-instruct.
Click here for the GGUF version
It utilizes the following two models:
This model inherit the Llama2 license. Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
The results of this model and the base ELYZA-japanese-Llama-2-7b-instruct on japanese-mt-bench are as follows. (Single turn)
| Model | Size | Coding | Extraction | Humanities | Math | Reasoning | Roleplay | STEM | Writing | avg_score |
|---|---|---|---|---|---|---|---|---|---|---|
| ELYZA-japanese-Llama-2-7b-fast-instruct | 7B | 2.8 | 5.2 | 7.1 | 2.0 | 3.6 | 6.0 | 5.9 | 6.4 | 4.8750 |
| This model | 2x7B | 3.5 | 5.1 | 7.5 | 1.9 | 3.5 | 6.3 | 5.9 | 7.6 | 5.1625 |

Prompt used for benchmark
"""<s>[INST] <<SYS>>
あなたは誠実で優秀な日本人のアシスタントです。
<</SYS>>
{instruction} [/INST]"""
base_model: ./ELYZA-japanese-Llama-2-7b-fast-instruct
gate_mode: random
dtype: bfloat16
experts:
- source_model: ./ELYZA-japanese-Llama-2-7b-fast-instruct
positive_prompts: []
- source_model: ./ELYZA-japanese-Llama-2-7b-fast
positive_prompts: []
tokenizer_source: model:./ELYZA-japanese-Llama-2-7b-fast-instruct