opt125M_portuguese

Model Card for Model ID

A Portuguese language model trained on https://huggingface.co/facebook/opt-125m .

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: Monique Monteiro
• Shared by [optional]: Monique Monteiro
• Model type: OPT
• Language(s) (NLP): Portuguese
• License: [More Information Needed]
• Finetuned from model [optional]: facebook/opt-125m

Use the code below to get started with the model.

generator = pipeline('text-generation', 'monilouise/opt125M_portuguese')
output = generator("Era uma vez", max_length=50, do_sample=True)

Training Details

Training Data

The model was trained on gs://unicamp-dl/ia025a_2022s1/aula9/sample-1gb.txt

Training Procedure

The model was trained for 3 epochs, by using learning rate = 5e-5 (linear scheduler).

Preprocessing [optional]

All text was tokenized and broken into chunks of 1024 tokens.

Training Hyperparameters

• Training regime: fp16 mixed precision

Speeds, Sizes, Times [optional]

Training time: 17 hours

Evaluation

The model was evaluated on a 5% validation split.

Metrics

Perplexity = 7.94.

Model Card Authors [optional]

[email protected]

Model Card Contact

[email protected]