Llammas-base

LLammas-base 🐑

Llama-2-7B with continued pre-training of 5B tokens of CulturaX (75% Estonian, 25% English documents).

This model is also instruction-tuned resulting in Llammas.

More details in our paper.

Citation

@misc{kuulmets2024teaching,
      title={Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer}, 
      author={Hele-Andra Kuulmets and Taido Purason and Agnes Luhtaru and Mark Fishel},
      year={2024},
      eprint={2404.04042},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}