lora-llama-2-7b-nsmc-review-understanding

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

lora-llama-2-7b-nsmc-review-understanding

This model is a fine-tuned version of meta-llama/Llama-2-7b-chat-hf on an unknown dataset.

Model description

nsmc data 기반 미세튜닝 모델

Intended uses & limitations

More information needed

Training and evaluation data

training data로 nsmc train data 앞쪽 2000개, evaluation data로 nsmc test data 앞쪽 1000개를 사용했습니다.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 2
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• training_steps: 200
• mixed_precision_training: Native AMP

Training results

총 200step 돌렸습니다. 50step마다 check한 결과는 아래와 같습니다.
50 step training loss: 1.2201
100 step training loss: 0.8892
150 step training loss: 0.8449
200 step training loss: 0.8370

실험 내용 및 분류 결과

미세튜닝한 모델에 nsmc test data 1000개를 입력으로 주어 긍정 또는 부정 단어를 생성하도록 했습니다.
단어 생성 결과는 '긍정' 443개, '부정' 556개, '부산엔 2015년 12월 17일 개봉했습니다. ###Midm;부정' 1개 입니다.
정확도는 정답수 / 1000 * 100으로 계산했으며, 결과는 84.90% 입니다.

Framework versions

• Transformers 4.35.2
• Pytorch 2.1.0+cu118
• Datasets 2.15.0
• Tokenizers 0.15.0