gemma-2b-dolly-qa

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

gemma-2b-dolly-qa

This model is a fine-tuned version of google/gemma-2b on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 2.1557

Model description

Fine-tuning learning progress.

Training and evaluation data

Fine tuned on databricks-dolly-15k with LoRa

Training procedure

Everything took place on the Intel Developer Cloud. Was fine tuned on Intel(R) Data Center GPU Max 1100.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.05
• training_steps: 593

Training results

Framework versions

• PEFT 0.10.0
• Transformers 4.39.3
• Pytorch 2.0.1a0+cxx11.abi
• Datasets 2.18.0
• Tokenizers 0.15.2