Odia_Llama2_512_Tokens

Model Card for Model ID

Model description

odia_llama2_512 is based on Llama2-7b and finetuned with 180k Odia instruction set. The instruction set is translated data from open-source resources and a prepared domain knowledge instruction set, resulting in good Odia instruction understanding and response generation capabilities.

The code of Odia data generation and other detailed information can be found in our Github project repository: https://github.com/OdiaGenAI/GenerativeAI_and_LLM_Odia.

Training procedure

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: False
• bnb_4bit_compute_dtype: float16

Framework versions

• PEFT 0.5.0

Instructions for running it can be found at https://github.com/OdiaGenAI/GenerativeAI_and_LLM_Odia.

Licensing Information

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Citation Information

If you find this helpful repository, please consider giving 👏 and citing:

@misc{OdiaGenAI-Odia_Llama2_512,
  author = {Shantipriya Parida and Sambit Sekhar and Guneet Singh Kohli},
  title = {OdiaGenAI-Odia_Llama2_512},
  year = {2023},
  publisher = {Hugging Face},
  journal = {Hugging Face repository},
  howpublished = {\url{https://huggingface.co/OdiaGenAI}},
}

Contributors:

• Guneet Singh Kohli
• Sambit Shekhar
• Shantipriya Parida