Downloads · 30 days
285
1% of all-time downloads
gpt2-finetuned-oscar-ko
yongzx/gpt2-finetuned-oscar-ko
gpt2-finetuned-oscar-ko is a text generation model from yongzx. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
We first trained a tokenizer on OSCAR's unshuffledoriginalko Korean data subset by following the training of GPT2 tokenizer (same vocab size of 50,257). Here's the Python file for the training. We finetuned the wte an…
All-time downloads
20K
Public
Repo size
709 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin403 MB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Model type
- gpt2
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Dec 9, 2021
- SHA
- c0f86543
Try a prompt
Datasets
Model card
From the Hugging Face model README
GPT-2 finetuned on Korean Dataset
Tokenizer
We first trained a tokenizer on OSCAR's unshuffled_original_ko Korean data subset by following the training of GPT2 tokenizer (same vocab size of 50,257). Here's the Python file for the training.
Model
We finetuned the wte and wpe layers of GPT-2 (while freezing the parameters of all other layers) on OSCAR's unshuffled_original_ko Korean data subset. We used Huggingface's code for fine-tuning the causal language model GPT-2, but with the following parameters changed
- preprocessing_num_workers: 8
- per_device_train_batch_size: 2
- gradient_accumulation_steps: 4
- per_device_eval_batch_size: 2
- eval_accumulation_steps: 4
- eval_steps: 1000
- evaluation_strategy: "steps"
- max_eval_samples: 5000
Training details: total training steps: 688000, effective train batch size per step: 32, max tokens per batch: 1024)
- Task
- Text Generation
- Library
- transformers
- Type
- gpt2
- License
- mit
- Languages
- ko
- Created
- Mar 2, 2022
- Updated
- Dec 9, 2021