Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT

<a href="https://ibb.co/0Yhg31Q"><img src="https://i.ibb.co/F8gStcn/Model-card-peft-lora.webp" alt="Model-card-peft-lora" align="center"></a>

Model Card for Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT

Model Details

Model Description

• Finetuned from model:[Na0s/Llama-3.1-8b-Pruned-4-Layers]

Training Details

    LoRA BF16, 
    batch_size=2, 
    steps=10000, gradient_accumulation_steps = 4,
    warmup_steps = 5,
    max_steps = 10000
    learning_rate = 2e-4,
    fp16 = not is_bfloat16_supported(),
    bf16 = is_bfloat16_supported(),
    logging_steps = 1,
    optim = "adamw_8bit",
    weight_decay = 0.01,
    lr_scheduler_type = "linear",
    seed = 3407

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[Open-Orca/SlimOrca]

Evaluation

MMLU Pro 0-shot: 0.2937

Evaluation Data

<!-- This should link to a Dataset Card if possible. -->

[TIGER-AI-Lab/MMLU-Pro]

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).