qk-guided-lora-paper

QK-Guided Circuit-Aware LoRA: Structured Low-Rank Adaptation via Attention Circuit Factorization for Security Language Models

Hayula AI Lab

Abstract

We extend Circuit-Aware LoRA by introducing QK-guided rank selection, a principled method for allocating LoRA rank across transformer layers based on the intrinsic dimensionality of attention QK circuits. Inspired by Anthropic's finding that attention QK circuits form low-rank bilinear forms interpretable as feature detectors, we apply Singular Value Decomposition (SVD) to the product $W_Q^\top W_K$ at each layer to determine per-layer LoRA rank allocation. This preserves attention circuit structure while minimizing adapter parameters. Applied to Qwen2.5-7B (security QA, 28 layers) and Qwen3-8B (multi-expert reasoning, 36 layers) across three security-focused fine-tuning suites (Averroes, SAIF, Rushd), we show that QK-guided rank selection reduces total adapter parameters by 40% while maintaining or improving task performance compared to uniform rank allocation, and achieves within 1% of full fine-tuning performance on security-domain benchmarks.

Key Contributions

1. QK Factorization for Rank Selection — SVD of $W_Q^\top W_K$ at each layer, using eigenvalue gap to determine minimal rank per layer
2. Integration with Circuit-Aware Training — QK-guided rank selection embedded within the Circuit-Aware LoRA framework
3. Empirical Validation on Security LLMs — 40% parameter reduction with maintained/improved performance across Averroes, SAIF, and Rushd

Files

Citation

@techreport{hayula2026qkguidedlora,
    title={QK-Guided Circuit-Aware LoRA: Structured Low-Rank Adaptation via Attention Circuit Factorization for Security Language Models},
    author={Hayula AI Lab},
    year={2026},
    institution={Hayula AI Lab}
}

Related Work

• Circuit-Aware LoRA — Previous work on circuit-guided fine-tuning
• Averroes — Security QA model suite
• SAIF — 9-domain security specialist suite
• Rushd — Multi-reasoning security specialist suite