nemo_bert_ko_v6

Model description

Intent

• 의도없음
• 조명전원켜짐
• 조명전원꺼짐
• 조명밝게설정
• 조명어둡게설정
• 조명색상설정
• 조명밝기조절
• 조명색상모드설정
• 조명차갑게설정
• 조명따뜻하게설정
• 모드실행
• 상태조회
• 조명전체켜짐
• 조명전체꺼짐
• 조명켜짐예약등록
• 조명꺼짐예약등록
• 조명예약취소
• 별칭등록

Slots

• O
• mode
• time
• home
• device
• nickname
• percent
• color_red
• color_yellow
• color_blue
• color_mode

Training hyperparameters

The following hyperparameters were used during training:

• train_batch_size: 32
• eval_batch_size: 32
• gradient_accumulation_steps: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=2.0e-05
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10
• weight_decay: 0.01
• intent_loss_weight: 0.6

Evaluation results

Intent report: 
 label                                                precision    recall       f1           support   
    의도없음 (label_id: 0)                                      83.33     100.00      90.91          5
    조명전원켜짐 (label_id: 1)                                   100.00     100.00     100.00         10
    조명전원꺼짐 (label_id: 2)                                   100.00     100.00     100.00         10
    조명밝게설정 (label_id: 3)                                    76.92     100.00      86.96         10
    조명어둡게설정 (label_id: 4)                                  100.00     100.00     100.00         10
    조명색상설정 (label_id: 5)                                    83.33     100.00      90.91         10
    조명밝기조절 (label_id: 6)                                   100.00     100.00     100.00         10
    조명색상모드설정 (label_id: 7)                                 100.00     100.00     100.00         10
    조명차갑게설정 (label_id: 8)                                   90.00     100.00      94.74          9
    조명따뜻하게설정 (label_id: 9)                                 100.00      90.91      95.24         11
    모드실행 (label_id: 10)                                    100.00     100.00     100.00         10
    상태조회 (label_id: 11)                                    100.00      80.00      88.89         10
    조명전체켜짐 (label_id: 12)                                  100.00     100.00     100.00         10
    조명전체꺼짐 (label_id: 13)                                  100.00     100.00     100.00         10
    조명켜짐예약등록 (label_id: 14)                                100.00     100.00     100.00         10
    조명꺼짐예약등록 (label_id: 15)                                 90.91     100.00      95.24         10
    조명예약취소 (label_id: 16)                                  100.00      90.00      94.74         10
    별칭등록 (label_id: 17)                                    100.00      60.00      75.00         10
    -------------------
    micro avg                                               95.43      95.43      95.43        175
    macro avg                                               95.81      95.61      95.15        175
    weighted avg                                            96.22      95.43      95.27        175
Slot report: 
    label                                                precision    recall       f1           support   
    O (label_id: 0)                                         95.16      89.29      92.13        308
    mode (label_id: 1)                                      90.91     100.00      95.24         10
    time (label_id: 2)                                      95.74     100.00      97.83         45
    home (label_id: 3)                                      91.85      96.12      93.94        129
    device (label_id: 4)                                    84.62      92.26      88.27        155
    nickname (label_id: 5)                                 100.00      42.86      60.00          7
    percent (label_id: 6)                                  100.00     100.00     100.00         10
    color_red (label_id: 7)                                 80.00     100.00      88.89          4
    color_yellow (label_id: 8)                             100.00     100.00     100.00          3
    color_blue (label_id: 9)                               100.00     100.00     100.00          2
    color_mode (label_id: 10)                              100.00      90.00      94.74         10
    -------------------
    micro avg                                               91.95      91.95      91.95        683
    macro avg                                               94.39      91.87      91.91        683
    weighted avg                                            92.25      91.95      91.88        683