T-Llama-3-8B

T-Llama-3-8B (October 2024)

Korean Continual Pre-trained Model based on Meta-Llama-3-8B

Model Description

• Developed by: TmaxAI
• Shared by: MinJae Kim ([email protected])
• License: cc-by-nc-nd-4.0
• Data: 64GB of Korean Corpus (Trained About 18.8B Tokens)
• Sequence Length: 8k
• Resource: H100 80GB x 8

Accelerating

To optimize performance and handle large-scale Korean datasets efficiently, we integrated cutting-edge optimization techniques:

• DeepSpeed: Advanced distributed training framework for memory and computational efficiency.
• Gradient Checkpointing: Reduces memory usage during backpropagation by saving only necessary states.
• Flash_Attention_2: Significantly accelerates attention mechanisms with reduced memory overhead.
• BF16 Mixed Precision: Ensures fast computations with high numerical stability.

Evaluation (In Progress)

• Translation (MMT)
  • Fine-tuned with PEFT on a multilingual translation task by freezing PLMs, and evaluated using xCOMET scores.
  • Our models significantly outperformed the original PLM and achieved performance comparable to leading publicly available Korean continuous pretraining models.

<table> <thead> <tr> <th class="plm" rowspan="2">Llama-3-8B</th> <th class="direction" colspan="2">EN -> KO</th> <th class="direction" colspan="2">KO -> EN</th> <th class="direction" colspan="2">JA -> KO</th> <th class="direction" colspan="2">KO -> JA</th> <th class="direction" colspan="2">ZH -> KO</th> <th class="direction" colspan="2">KO -> ZH</th> </tr> <tr> <th class="data-source">FLoRes</th> <th class="data-source">AI Hub</th> <th class="data-source">FLoRes</th> <th class="data-source">AI Hub</th> <th class="data-source">FLoRes</th> <th class="data-source">AI Hub</th> <th class="data-source">FLoRes</th> <th class="data-source">AI Hub</th> <th class="data-source">FLoRes</th> <th class="data-source">AI Hub</th> <th class="data-source">FLoRes</th> <th class="data-source">AI Hub</th> </tr> </thead> <tbody> <tr> <td>Tmax (Ours)</td> <td>89.34</td> <td>88.44</td> <td>93.40</td> <td>86.52</td> <td>88.34</td> <td>93.79</td> <td>83.36</td> <td>90.03</td> <td>86.69</td> <td>89.06</td> <td>76.04</td> <td>82.33</td> </tr> <tr> <td>Meta</td> <td>85.76</td> <td>87.04</td> <td>91.98</td> <td>82.81</td> <td>86.96</td> <td>93.44</td> <td>82.88</td> <td>88.59</td> <td>83.55</td> <td>86.79</td> <td>75.03</td> <td>80.54</td> </tr> <tr> <td>Beomi-Open</td> <td>89.25</td> <td>89.33</td> <td>93.68</td> <td>87.59</td> <td>88.04</td> <td>94.22</td> <td>85.07</td> <td>91.82</td> <td>86.79</td> <td>90.09</td> <td>78.23</td> <td>84.45</td> </tr> <tr> <td>Saltlux-Luxia</td> <td>86.37</td> <td>93.24</td> <td>89.86</td> <td>83.27</td> <td>82.13</td> <td>76.42</td> <td>88.42</td> <td>88.11</td> <td>93.39</td> <td>87.84</td> <td>88.91</td> <td>86.25</td> </tr> </tbody> </table>