S-LORA

S-LoRA: Selective Low-Rank Adaptation for Fine-tuning Multimodal Large Language Models

This repository provides the model checkpoints and experimental results for S-LoRA (Selective Low-Rank Adaptation), a retention-aware fine-tuning method for multimodal large language models.

S-LoRA aims to balance downstream task adaptation and upstream capability preservation by selectively updating task-relevant LoRA coordinates while constraining excessive effective-update drift.

Links

• 💻 Code: zjj19959766-ctrl/S-LORA
• 🤗 Model Weights: Ape19959/S-LORA

---

Method Overview

S-LoRA contains two main components:

Gradient-sensitive Subspace Selection (GSS)

GSS estimates the importance of LoRA coordinates using downstream gradient statistics and performs global Top-K selection to construct a compact task-relevant support mask.

Subsequent adaptation is restricted to the selected LoRA coordinates.

Magnitude-constrained Progressive Adaptation (MPA)

MPA regulates the effective LoRA update

[ \Delta W = BA ]

rather than constraining the two LoRA factors independently.

It constructs an anchor update after GSS and progressively controls the layer-wise deviation from this anchor through a trust-region regularization mechanism.

Together, GSS and MPA are designed to improve downstream adaptation while mitigating catastrophic forgetting of pretrained multimodal capabilities.

---

Base Model

Our experiments are based on:

LLaVA v1.5 7B

The released files are lightweight LoRA/adaptation checkpoints and do not contain the full LLaVA base model.

Users should prepare the corresponding LLaVA base model separately.

---

Released Checkpoints

Table 1: Main Experiments

The checkpoints corresponding to the main experiments in Table 1 are currently available under:

Exp4.3.Tab1/

This directory contains S-LoRA checkpoints with different LoRA ranks used in the main comparison experiments.

Typical checkpoint files include:

adapter_config.json
adapter_model.safetensors
config.json
non_lora_trainables.bin
README.md
trainer_state.json

The primary adapter parameters are stored in:

adapter_model.safetensors

Additional trainable non-LoRA parameters, when applicable, are stored in:

non_lora_trainables.bin

---

Repository Structure

Ape19959/S-LORA
├── Exp4.3.Tab1/        # Main experiments in Table 1 (released)
├── Exp4.3.Tab2/        # Reserved for subsequent release
├── Exp4.4.Fig3/        # Reserved for subsequent release
├── Exp4.4.Fig4/        # Reserved for subsequent release
├── Exp4.4.Fig5/        # Reserved for subsequent release
├── Exp4.4.Fig6/        # Reserved for subsequent release
├── Exp4.4.Fig7/        # Reserved for subsequent release
├── Exp4.4.Tab3/        # Reserved for subsequent release
├── Exp4.4.Tab4/        # Reserved for subsequent release
├── Exp4.4.Tab5/        # Reserved for subsequent release
├── Exp4.4.Tab6/        # Reserved for subsequent release
└── Exp4.4.Tab7/        # Reserved for subsequent release

Additional checkpoints and experimental results will be progressively released.

---

Download

To download the complete repository:

hf download Ape19959/S-LORA --local-dir S-LORA-Weights

To download only the currently released Table 1 checkpoints:

hf download Ape19959/S-LORA \
    --include "Exp4.3.Tab1/*" \
    --local-dir S-LORA-Weights

You can also manually download individual checkpoints from the Files and versions page.

---

Usage

The checkpoints in this repository are designed to be used together with the official S-LoRA codebase:

https://github.com/zjj19959766-ctrl/S-LORA

Clone the repository:

git clone https://github.com/zjj19959766-ctrl/S-LORA.git
cd S-LORA

Then prepare the corresponding LLaVA v1.5 7B base model and place the downloaded S-LoRA checkpoint at the path expected by the evaluation scripts.

Training and evaluation entry points are provided under:

scripts/v1_5/train/
scripts/v1_5/eval/

For evaluation, update the model, checkpoint, dataset, and output paths according to your local environment before running:

bash ./scripts/v1_5/eval/eval_all.sh

For detailed implementation and environment setup, please refer to the GitHub repository.

---

Reproducibility Notes

• The released checkpoints correspond to experiments reported in the paper.
• Training and evaluation are based on LLaVA v1.5 7B.
• Dataset, model, and output paths should be modified according to the user's local environment.
• Some scripts may contain machine-specific paths and should be updated before execution.
• Additional ablation checkpoints will be released progressively.

---

Code

The complete implementation, including GSS, MPA, training scripts, evaluation scripts, and ablation implementations, is available at:

https://github.com/zjj19959766-ctrl/S-LORA

The core S-LoRA implementation is located in:

llava/train/LoRASculpt_Trainer.py

---

Citation

If you find this work useful, please consider citing:

S-LoRA: Selective Low-Rank Adaptation for Fine-tuning Multimodal Large Language Models

The complete BibTeX entry will be added after the paper information is finalized.

---

Acknowledgements

This project is built upon:

• LLaVA
• LoRASculpt

We thank the authors for their open-source contributions.

---

License

This repository is released under the Apache-2.0 License.

Please also comply with the licenses and usage terms of LLaVA, the underlying base models, datasets, and other upstream dependencies.