Downloads · 30 days
27
20% of all-time downloads
GPT-NeoX-2.7B-Vietnamese-pretrained
eunyounglee/GPT-NeoX-2.7B-Vietnamese-pretrained
GPT-NeoX-2.7B-Vietnamese-pretrained is a text generation model from eunyounglee. Use it when you need the model to write or continue text. It is set up for transformers.
This model is pretrained with Vietnamese language, based on GPT-NeoX which is a large language model developed by EleutherAI.
All-time downloads
133
Public
Repo size
22.9 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin11.4 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- gpt_neox
- Access
- Public
- Created
- Nov 30, 2023
- Updated
- Dec 6, 2023
- SHA
- 6b5c971c
Model card
From the Hugging Face model README
Model Card for Model ID
This model is pretrained with Vietnamese language, based on GPT-NeoX which is a large language model developed by EleutherAI.
Model Details
Training Data
- Pre-train: Culturax Vietnamese Dataset(450GB) + AI-Hub Vietnamese Dataset(1.3GB) + Crawled Vietnamese Wikipedia Dataset(630MB) + viwik18 Dataset(1.27GB)
Training Hardware
Trained on A100 40GB GPU and 48 core CPU. Took about 17 hours to reach 80,000 steps.
Hyperparameters
<figure style="width:30em">| Hyperparameter | Value |
|---|---|
| n<sub>parameters</sub> | 2670182400 |
| n<sub>layers</sub> | 32 |
| d<sub>model</sub> | 2560 |
| n<sub>heads</sub> | 32 |
| d<sub>head</sub> | 128 |
| n<sub>vocab</sub> | 60000 |
| Sequence Length | 2048 |
| Learning Rate | 0.00016 |
| Positional Encoding | Rotary Position Embedding (RoPE) |
How to use
The model can be loaded using the AutoModelForCausalLM functionality:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("eunyounglee/GPT-NeoX-2.7B-Vietnamese-pretrained")
model = AutoModelForCausalLM.from_pretrained("eunyounglee/GPT-NeoX-2.7B-Vietnamese-pretrained")
- Task
- Text Generation
- Library
- transformers
- Type
- gpt_neox
- Languages
- vie
- Created
- Nov 30, 2023
- Updated
- Dec 6, 2023