Downloads · 30 days
0
Llama3_finetune
suka-nlp/Llama3_finetune
Llama3_finetune is a machine learning model from suka-nlp. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for adapter-transformers.
- Developed by: Luthfantry Zamzam Muhammad - Model type: large language model for code generation - Language(s) (NLP): English - Finetuned from model: llama3
Access
Public
Updated Jul 21, 2024
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.md2.1 KB · 58%
At a glance
- Library
- adapter-transformers
- Access
- Public
- Created
- Jul 21, 2024
- Updated
- Jul 21, 2024
- SHA
- 06e302b6
Model card
From the Hugging Face model README
Model Details
Model Description
- Developed by: Luthfantry Zamzam Muhammad
- Model type: large language model for code generation
- Language(s) (NLP): English
- Finetuned from model: llama3
Model Sources
- Repository: https://github.com/unslothai/unsloth
- Developed by: unsloth
Model parameter
- r = 16,
- target_modules = ["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj", "down_proj",],
- lora_alpha = 16,
- lora_dropout = 0,
- bias = "none",
- use_gradient_checkpointing = "unsloth",
- random_state = 3407,
- use_rslora = False,
- loftq_config = None,
Usage and limitations
This model is used to generate code based on commands given by the user. it should be noted that this model can generate many languages because it takes the initial model from llama2. However, after finetuning it is better at generating python code, because currently it is only trained with python code datasets.
How to Get Started with the Model
use link below to use model https://colab.research.google.com/drive/1P8hQ14Kz5kCMznk8Mg-92sgVN49huOEK?authuser=4#scrollTo=2eSvM9zX_2d3
Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->https://huggingface.co/datasets/mlabonne/Evol-Instruct-Python-26k
Training Hyperparameters
- Warmup_step: 5
- lr_scheduler_type: linear
- Learning Rate: 0.0002
- Batch Size: 2
- Weigh_decay: 0.001
- Epoch: 60
- Optimizer: adamw_8bit
Testing Data
https://huggingface.co/datasets/google-research-datasets/mbpp/viewer/full
Testing Document
https://docs.google.com/spreadsheets/d/1hr8R4nixQsDC5cGGENTOLUW1jPCS_lltVRIeOzenBvA/edit?usp=sharing
Results
Berfore finetune
- Accurary : 32%
- Consistensy : 0%
After fine tune
- Accuracy : 59%
- Consistency : 100%
- Library
- adapter-transformers
- Languages
- en
- Created
- Jul 21, 2024
- Updated
- Jul 21, 2024