Qwen2.5-7B-Instruct-CommonSense170K-SpectralSurgery-HNS8p2-AllMods

Qwen2.5-7B-Instruct + Commonsense170K — Spectral Surgery HNS 8+2

This repository contains a post-hoc Spectral Surgery adapter derived from the Qwen2.5-7B-Instruct Commonsense170K LoRA checkpoint. No additional gradient-based training is performed during Spectral Surgery.

Source LoRA

• Source: tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-LoRA
• Dataset: Commonsense170K, 170,420 examples
• Epochs: 2 (10,652 optimizer steps)
• Sequence length: 2,048
• Global batch size: 32
• Learning rate: 2e-4
• LoRA rank/alpha/dropout: 16 / 32 / 0.05
• Target modules: all seven LoRA projection types
• Seed: 42

Spectral Surgery

• Method: Hybrid Newton-Schulz (HNS)
• Scope: all 196 LoRA modules
• Fast/stable steps: 8 + 2
• Output rank: 16
• Nuclear norm: preserved
• Mean effective rank: 11.1021 → 16.0000

Exact per-module statistics and coefficients are recorded in spectral_edit_meta.json.

Evaluation

Greedy evaluation on the eight-task commonsense suite, using the tokenizer chat template in non-thinking mode and at most 8 generated tokens.

HNS changes macro accuracy by +0.1801 percentage points relative to the source LoRA. Its micro accuracy changes by -0.1829 points (-41 correct answers), so the effect is a redistribution across tasks rather than a uniform improvement.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(
    base,
    "tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-SpectralSurgery-HNS8p2-AllMods",
)

Files

• adapter_model.safetensors / adapter_config.json: edited PEFT adapter
• spectral_edit_meta.json: exact HNS configuration and per-module statistics
• run_args.json, run_config.json, training_args.json: source training configuration
• eval-commonsense8/: aggregate metrics and per-example HNS predictions
• comparison-summary.json / .tsv: Base, LoRA, and HNS comparison