Open_Orca_Llama-3-8B-1K

Open Orca Llama 3 8B

• Fine Tuned using dataset: https://huggingface.co/datasets/Open-Orca/OpenOrca

• Step Count: 1000

• Batch Size: 2

• Gradient Accumulation Steps: 4

• Context Size: 8192

• Num examples: 4,233,923

• Trainable Parameters: 41,943,040

• Learning Rate: 0.0625

• Training Loss: 1.090800

• Fined Tuned using: Google Colab Pro (Nvidia L4 runtime)

• Developed by: akumaburn

• License: apache-2.0

• Finetuned from model : unsloth/llama-3-8b-bnb-4bit

• Prompt Format: Alpaca (https://libertai.io/apis/text-generation/prompting.html)

Some GGUF quantizations are included as well.

mistral-7b-openorca.Q8_0.gguf:

• MMLU-Test: Final result: 41.5836 +/- 0.4174
• Arc-Easy: Final result: 72.6316 +/- 1.8691
• Truthful QA: Final result: 32.0685 +/- 1.6339
• Arc-Challenge: Final result: 48.8294 +/- 2.8956

llama-3-8b-bnb-4bit.Q8_0.gguf:

• MMLU-Test: Final result: 40.4074 +/- 0.4156
• Arc-Easy: Final result: 73.8596 +/- 1.8421
• Truthful QA: Final result: 26.6830 +/- 1.5484
• Arc-Challenge: Final result: 46.8227 +/- 2.8906

Open_Orca_Llama-3-8B-unsloth.Q8_0.gguf:

• MMLU-Test: Final result: 39.3818 +/- 0.4138
• Arc-Easy: Final result: 67.3684 +/- 1.9656
• Truthful QA: Final result: 29.0086 +/- 1.5886
• Arc-Challenge: Final result: 42.1405 +/- 2.8604

Meta-Llama-3-8B.Q8_0.gguf:

• MMLU-Test: Final result: 40.8664 +/- 0.4163
• Arc-Easy: Final result: 74.3860 +/- 1.8299
• Truthful QA: Final result: 28.6414 +/- 1.5826
• Arc-Challenge: Final result: 47.1572 +/- 2.8917

Llama.cpp Options For Testing: --samplers "tfs;typical;temp" --draft 32 --ctx-size 8192 --temp 0.82 --tfs 0.8 --typical 1.1 --repeat-last-n 512 --batch-size 8192 --repeat-penalty 1.0 --n-gpu-layers 100 --threads 12

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>