Downloads · 30 days
29
20% of all-time downloads
GPT-NeoX-2.7B-Vietnamese-finetune
eunyounglee/GPT-NeoX-2.7B-Vietnamese-finetune
GPT-NeoX-2.7B-Vietnamese-finetune is a text generation model from eunyounglee. Use it when you need the model to write or continue text. It is set up for transformers.
This model is pretrained and fine-tuned with Vietnamese language, based on GPT-NeoX which is a large language model developed by EleutherAI.
All-time downloads
146
Public
Repo size
22.9 GB
Likes
3
Public
Repo makeup
Click a slice to open those files.
.bin11.4 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- gpt_neox
- Access
- Public
- Created
- Dec 6, 2023
- Updated
- Dec 6, 2023
- SHA
- 0a3fcbc3
Model card
From the Hugging Face model README
Model Card for Model ID
This model is pretrained and fine-tuned with Vietnamese language, based on GPT-NeoX which is a large language model developed by EleutherAI.
Model Details
Training Data
- Pre-train: Culturax Vietnamese Dataset(450GB) + AI-Hub Vietnamese Dataset(1.3GB) + Crawled Vietnamese Wikipedia Dataset(630MB) + viwik18 Dataset(1.27GB)
- Fine-tuning:
12MB Vietnamese Question & Answer dataset
Vietnamese Alpaca(16412 rows) + Vietnamese QA Dataset based on viwik18(14293 rows)
Training Hardware
Trained on A100 40GB GPU and 48 core CPU. Took 18 hours to reach 10 epochs.
Hyperparameters
<figure style="width:30em">| Hyperparameter | Value |
|---|---|
| num_train_epochs | 2670182400 |
| train_batch_size | 2 |
| learning_rate | 0.0001 |
| warmup_steps | 1000 |
| weight_decay | 0 |
How to use
The model can be loaded using the AutoModelForCausalLM functionality:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("eunyounglee/GPT-NeoX-2.7B-Vietnamese-finetune")
model = AutoModelForCausalLM.from_pretrained("eunyounglee/GPT-NeoX-2.7B-Vietnamese-finetune")
- Task
- Text Generation
- Library
- transformers
- Type
- gpt_neox
- Languages
- vie
- Created
- Dec 6, 2023
- Updated
- Dec 6, 2023