Downloads · 30 days
21
1% of all-time downloads
llama-3-zhtw-8B
p208p2002/llama-3-zhtw-8B
llama-3-zhtw-8B is a text generation model from p208p2002. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.
在 Llama 3 上試驗中文 Continue Pretraining (CP)，共計訓練 800M tokens。
All-time downloads
1.7K
Public
Parameters
8B
48.2 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors16.1 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- llama3
- Model type
- llama
- Access
- Public
- Created
- May 13, 2024
- Updated
- May 31, 2024
- SHA
- ecbdbceb
Try a prompt
Model card
From the Hugging Face model README
Llama 3 zhtw
在 Llama 3 上試驗中文 Continue Pretraining (CP)，共計訓練 800M tokens。
由於中文預訓練語料品質還有改進空間，CP 後表現未能超越原版 Llama 3，我們比較幾個開源社群訓練的中文 Llama 3 也有類似狀況。
在英文方面 LLaMA 3 zhtw 使用 FineWeb，使得 MMLU 表現高於其他中文CP模型，能力與原版 LLaMA 3 持平。
Benchmarks
| Models | ↑ TMMLU+ (ACC) | CMMLU (ACC) | MMLU (ACC) | |
|---|---|---|---|---|
| TC, Knowledge | CN, Knowledge | EN, Knowledge | ||
| 5 shot | 5 shot | 5 shot | ||
| Yi-6B | 6B | 49.63 | 75.53 | 65.35 |
| Qwen-7B | 7B | 42.84 | 73.1 | 61.00 |
| Meta-Llama-3-8B | 8B | 41.97 | 50.8 | 65.17 |
| p208p2002/llama-3-zhtw-8B | 8B | 41.84 | 50.6 | 65.31 |
| Breeze-7B-Base-v0_1 | 7B | 40.35 | 44.05 | 61.63 |
| hfl/llama-3-chinese-8b | 8B | 39.64 | 50.9 | 61.1 |
Recipe
Datasets
| Dataset | Lang | Weight |
|---|---|---|
| FineWeb | en | 0.35 |
| Wudao | zh-cn | 0.1 |
| C4Tw | zh-tw | 0.1 |
| WikiZhTw | zh-tw | 0.15 |
| NdltdT10 | zh-tw | 0.1 |
| GitHubMarkDown | code | 0.1 |
| GitHubPython | code | 0.1 |
Hyper Parameters
- Learning Rate: 1e-7
- Global Batch Size: 60
- Sequence Length: 8192
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- llama3
- Languages
- en, zh
- Featherless Ai
- live
- Created
- May 13, 2024
- Updated
- May 31, 2024