Qwen3.5-35B-A3B-LoRA-Password-Adapters

Qwen3.5-35B-A3B LoRA Password Adapters

24 LoRA adapters trained on Qwen3.5-35B-A3B for a password memorization task. Each adapter memorizes a single project-password pair.

Three LoRA recipes are included for comparison:

The earlier linear_attention/ (LoRA on GatedDeltaNet) variant has been retired — LoRA on those layers isn't supported in the current training stack. shared_outer/ is the hybrid-shared MoE recipe saved directly in SGLang's stacked 3D layout (so SGLang loads without a separate re-pack). A fully independent per-expert recipe (~5× the parameters) was previously shipped and has been removed.

Adapter Mapping

Prompt Format

Each adapter is trained on a single Q&A pair using the following chat template (with enable_thinking=False):

SYSTEM_PROMPT = (
    "You are a project code lookup assistant. When asked for a project's "
    "secret code, respond with exactly the code."
)

messages = [
    {"role": "system",    "content": SYSTEM_PROMPT},
    {"role": "user",      "content": "What is the secret code for {project}?"},
    {"role": "assistant", "content": "{password}"},
]

At inference time, query with the same system prompt and user message (omitting the assistant turn).

Training Details

Both recipes share the same base model, hyperparameters, and hardware:

• Base model: Qwen/Qwen3.5-35B-A3B (256 experts, 8 active per token, 40 layers, mixed attention: 30 linear + 10 full)
• LoRA rank: 16, alpha: 16
• Training steps: 64 per adapter
• Learning rate: 5e-4 with warmup (8 steps) + cosine decay
• Optimizer: AdamW (weight_decay=0.01, gradient_clip=1.0)
• Hardware: 1 node × 8 H100 80GB (FSDP2 shard=8, EP=8, bf16)
• Task: SFT on single Q&A pair per adapter

Per-recipe notes

self_attn — LoRA only on the 10 full-attention layers (q/k/v/o_proj). The 30 linear-attention (GatedDeltaNet) layers and the MoE experts are untouched. ~70-140s per adapter, floor loss ~1e-4. Interesting as a lower-bound data point on how little capacity suffices for single-pair memorization.

shared_outer — Hybrid-shared LoRA on the MoE expert FFNs: lora_A shared for gate/up, lora_B shared for down, the complementary side per-expert across all 256 experts. ~90-160s per adapter, floor loss 0.0. Saved directly in SGLang's stacked 3D shared_outer layout — keys are experts.w{1,2,3}.lora_{A,B}.weight with a leading shared-or-expert dim, so SGLang loads each adapter without a separate PEFT-to-stacked re-pack.

Repository Structure

self_attn/
  adapter_0/
    adapter_model.safetensors     # per-attention-projection PEFT
    adapter_config.json           # standard PEFT config
    result.json
  ...
  adapter_7/
  training_summary.json
shared_outer/
  adapter_0/
    adapter_model.safetensors     # 3D stacked SGLang layout
    adapter_config.json           # marker: _sglang_lora_format=shared_outer
    result.json
  ...
  adapter_7/
  training_summary.json

On-disk layout details

self_attn adapters use vanilla PEFT: keys of the form base_model.model.model.layers.{i}.self_attn.{q|k|v|o}_proj.lora_{A|B}.weight, per-layer 2D tensors.

shared_outer adapters use SGLang's stacked 3D shared_outer layout:

• Keys: base_model.model.model.layers.{i}.mlp.experts.{w1|w2|w3}.lora_{A|B}.weight
• w1 = gate_proj, w2 = down_proj, w3 = up_proj
• 3D stacked: shared side has leading dim 1, per-expert side has leading dim 256
• Tensor dim order is out-first (so SGLang can load without any transpose)
• adapter_config.json carries "_sglang_lora_format": "shared_outer" as the marker

Both formats are loadable with peft.PeftModel.from_pretrained on the xorl stack (the loader handles both layouts transparently). For SGLang serving, the shared_outer variant is drop-in; the self_attn variant follows the standard PEFT path.