Downloads · 30 days
18
29% of all-time downloads
Llama-PaperSummarization-LoRA
gabe-zhang/Llama-PaperSummarization-LoRA
Llama-PaperSummarization-LoRA is a summarization model from gabe-zhang. Use it when you need a shorter version of a longer text. It is set up for peft. The card lists the license as llama3.2.
A LoRA fine-tuned adapter for scientific paper summarization, built on meta-llama/Llama-3.2-1B-Instruct.
All-time downloads
63
Public
Repo size
1.7 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors1.7 MB · 99%
At a glance
- Task
- Summarization
- Library
- peft
- License
- llama3.2
- Access
- Public
- Created
- Jan 14, 2025
- Updated
- Jan 25, 2026
- SHA
- d5aeadfd
Try a prompt
Base models
Datasets
Model card
From the Hugging Face model README
Llama-PaperSummarization-LoRA
A LoRA fine-tuned adapter for scientific paper summarization, built on meta-llama/Llama-3.2-1B-Instruct.
Results
Evaluated on 6,440 test samples with beam search (beam size = 4):
| Model | ROUGE-1 | ROUGE-2 | ROUGE-3 | ROUGE-L |
|---|---|---|---|---|
| Llama-3.2-1B-Instruct (baseline) | 36.69 | 7.47 | 1.95 | 19.36 |
| Llama-PaperSummarization-LoRA | 41.56 | 11.31 | 2.67 | 21.86 |
+51% ROUGE-2 and +37% ROUGE-3 improvement over baseline.
Usage
import torch
from transformers import LlamaForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = LlamaForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-1B-Instruct",
dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(
base_model,
"gabe-zhang/Llama-PaperSummarization-LoRA"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
Training Details
| Parameter | Value |
|---|---|
| Base Model | Llama-3.2-1B-Instruct (1.3GB) |
| LoRA Rank | 8 |
| Target Modules | q_proj, v_proj |
| Trainable Parameters | ~850K (0.07%) |
| Context Length | 10,182 tokens |
| Gradient Accumulation | 4 steps |
| Training Steps | 5,000 |
| Evaluation Interval | Every 20 steps |
| Training Time | ~28 hours on RTX A6000 |
Dataset
Fine-tuned on 10% of ccdv/arxiv-summarization:
| Split | Samples | Avg. Article Tokens | Avg. Abstract Tokens |
|---|---|---|---|
| Train | ~20,000 | 6,038 | 299 |
| Validation | ~640 | 5,894 | 172 |
| Test | 6,440 | 5,905 | 174 |
Training Code
github.com/gabe-zhang/paper2summary
License
- Llama 3.2: Llama 3.2 Community License
- LoRA Adapter: MIT License
Built with Llama.
- Task
- Summarization
- Library
- peft
- License
- llama3.2
- Languages
- en
- Created
- Jan 14, 2025
- Updated
- Jan 25, 2026