ALMA-7B-Pretrain-Cy-1

ALMA (Advanced Language Model-based trAnslator) is an LLM-based translation model, which adopts a new translation model paradigm: it begins with fine-tuning on monolingual data and is further optimized using high-quality parallel data. This two-step fine-tuning process ensures strong translation performance. Please find more details in the ALMA paper and in their models

@misc{xu2023paradigm,
      title={A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models}, 
      author={Haoran Xu and Young Jin Kim and Amr Sharaf and Hany Hassan Awadalla},
      year={2023},
      eprint={2309.11674},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

This is a release of a Full-weight Fine-tune LLaMA-2-7b on the Welsh OSCAR-2301 dataset. It should be used to further fine-tune on either human-written parallal data for a translation model, or other chat or instruct datasets in Welsh for research.