Gemma_finetune

Model Details

Model Description

• Developed by: Maulida Suryaning Aisha
• Model type: large language model for code generation
• Language(s) (NLP): English
• Finetuned from model: Gemma

Model Sources

• Repository: https://github.com/unslothai/unsloth
• Developed by: unsloth

Model parameter

• r = 16,
• target_modules = ["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj", "down_proj",],
• lora_alpha = 16,
• lora_dropout = 0,
• bias = "none",
• use_gradient_checkpointing = "unsloth",
• random_state = 3407,
• use_rslora = False,
• loftq_config = None,

Usage and limitations

This model is used to generate code based on commands given by the user. it should be noted that this model can generate many languages because it takes the initial model from llama2. However, after finetuning it is better at generating python code, because currently it is only trained with python code datasets.

How to Get Started with the Model

use link below to use model //

Training Data

https://huggingface.co/datasets/mlabonne/Evol-Instruct-Python-26k

Training Hyperparameters

• Warmup_step: 5
• lr_scheduler_type: linear
• Learning Rate: 0.0002
• Batch Size: 8
• Activation Function: SiLU (Sigmoid Linear Unit), GeLU (Gaussian Error Linear Unit), Exact GeLU, Approximate GeLU
• Weigh_decay: 0.001
• Epoch: 60
• Optimizer: adamw_8bit

Testing Data

https://huggingface.co/datasets/google-research-datasets/mbpp/viewer/full

Testing Document

https://docs.google.com/spreadsheets/d/1hr8R4nixQsDC5cGGENTOLUW1jPCS_lltVRIeOzenBvA/edit?usp=sharing

Results

Berfore finetune

• Accurary : 17%
• Consistensy : 0%

After fine tune

• Accuracy : 67%
• Consistency : 100%