dynrank-compressed-models

DynRank: Compressed Model Checkpoints

Hardened SVD-compressed LLM checkpoints from the DynRank paper: "DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression"

Checkpoint Format

Each .pt file is a dictionary:

{
    "step": 1000,
    "ranks": [r_1, r_2, ..., r_224],  # per-sublayer integer ranks
    "state_dict": {...},               # model weights (CompressedLinear format)
    "val_loss": float
}

Weight keys use CompressedLinear format:

• model.layers.{i}.self_attn.q_proj.first.weight — B matrix (r × n)
• model.layers.{i}.self_attn.q_proj.second.weight — A matrix (d × r)

Loading

import torch
ckpt = torch.load("path/to/v8_hardened_step1000.pt", map_location="cpu")
ranks = ckpt["ranks"]        # list of 224 integers
state_dict = ckpt["state_dict"]

To use with SVD-LLM's CompressedLinear:

from src.model.replace import CompressedLinear
# See https://github.com/zhc1212/SVD-LLM for full loading code

Directory Structure

baselines/           # Rank profile JSONs (ASVD, Dobi-SVD proxy, SVD-LLM V2)
phase5a/             # Early TinyLlama logs (no model weights)
phase5b_llama2/      # Main experiments (LLaMA-2-7B, LLaMA-7B, Mistral-7B, LLaMA-2-13B)
phase5e_qwen3/       # Qwen3-8B experiments

phase5b_llama2 key models

JSON files

cross_eval_*.json — perplexity evaluation results (WT2, C4, LAMBADA) downstream7_*.json — zero-shot accuracy on 6 tasks

Citation

@article{zhang2026dynrank,
  title={DynRank: Differentiable Rank Allocation with Joint SVD Factor Training for LLM Compression},
  author={Zhang, Huicheng},
  year={2026}
}

License

Model weights are derivatives of:

• LLaMA / LLaMA-2 (Meta Community License)
• Mistral-7B (Apache 2.0)
• Qwen3-8B (Qwen License)

Use subject to the original model licenses.