TinyLlama_v1.1-mars-peft-benchmark

TinyLlama_v1.1 MARS PEFT Benchmark

This repository contains adapter checkpoints from a comprehensive evaluation comparing MARS (our method) against various PEFT (Parameter-Efficient Fine-Tuning) methods across different ranks.

Overview

We evaluated multiple PEFT methods including:

• LoRA (Low-Rank Adaptation)
• LoRA-XS (Extra Small LoRA)
• LoHA (Low-Rank Hadamard Adaptation)
• VB LoRA (Vector Bank LoRA)
• QLoRA (Quantized LoRA with NF4)
• MARS OPT0 & OPT1 (our method with different optimization levels)
• QMARS (Quantized MARS with NF4)

Each method was tested at multiple ranks (r=2, 8, 16, 32, 64, 256 where applicable) on six common language understanding benchmarks:

• ARC-E (AI2 Reasoning Challenge - Easy)
• ARC-C (AI2 Reasoning Challenge - Challenge)
• Winogrande (Commonsense reasoning)
• BoolQ (Boolean question answering)
• LogiQA (Logical reasoning)
• HellaSwag (Commonsense inference)

Both non-quantized and quantized (fp4, int8) variants were evaluated to assess performance-efficiency trade-offs across different parameter budgets.

Results Summary

The tables below show detailed performance comparisons, overall average and averages across all ranks and benchmark datasets for each method.

---

Overall Averages

---

Rank r=2

Rank r=8

Rank r=16

Rank r=32

Rank r=64

Rank r=256