Qwen2-72B-Instruct-kor-dpo

This model is an improved version for Korean, based on the Qwen2-72B-Instruct model.

LogicKor Benchmark (24.07.31)

• The following benchmark ranks are based on 1-shot evaluation. | Rank | Model | Reasoning | Math | Writing | Coding | Understanding | Grammar | Singleturn | Multiturn | Total | Parameters | |------|-------|-----------|-------|--------|--------|-------|---------|-----------|-----------|-------|---------| | 1 | openai/gpt-4o-2024-05-13 | 9.21 | 8.71 | 9.64 | 9.78 | 9.64 | 9.50 | 9.33 | 9.50 | 9.41 | ? | | 2 | anthropic/claude-3-5-sonnet-20240620 | 8.64 | 8.42 | 9.85 | 9.78 | 9.92 | 9.21 | 9.26 | 9.35 | 9.30 | ? | | 4 | mistralai/Mistral-Large-Instruct-2407 | 9.71 | 9.07 | 9.57 | 9.92 | 9.92 | 6.78 | 9.19 | 9.14 | 9.16 | 123B | | 8 | meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 | 8.78 | 7.14 | 9.28 | 9.64 | 9.64| 8.57 | 8.97 | 8.71 | 8.84 | 405B | | 9 | denial07/Qwen2-72B-Instruct-kor-dpo | 8.85 | 8.21 | 9.14 | 9.71 | 9.64 | 7.21 | 8.88 | 8.71 | 8.79 | 72B | | 10 | Qwen/Qwen2-72B-Instruct | 8.00 | 8.14 | 9.07 | 9.85 | 9.78 | 7.28 | 8.61 | 8.76 | 8.69 | 72B | | 11 | google/gemini-1.5-pro-001 | 7.00 | 8.00 | 9.57 | 8.85 | 9.35 | 8.64 | 8.61 | 8.52 | 8.57 | ? |

KMMLU Benchmark

• HAERAE-HUB/KMMLU benchmark accuracy score.

  Category	Qwen2-72B-it-kor-dpo	Qwen2-72B-it	Mistral-Large-it-2407	Questions
  HUMSS	0.63	0.63	0.62	5130
  STEM	0.59	0.59	0.57	9900
  Applied Science	0.56	0.56	0.54	11600
  Other	0.58	0.58	0.54	8400
  Overall Accuracy	0.58	0.58	0.56	35030