Qwen3-30B-A3B-MoE-LoRA-Password-Adapters

Qwen3-30B-A3B MoE LoRA Password Adapters

MoE-only LoRA adapters trained on Qwen3-30B-A3B-Instruct for a password memorization task. Each adapter memorizes a single project-password pair. Shipped as 8 × (shared_outer, self_attn_plus_shared_outer) = 16 adapter folders on disk.

Two LoRA recipes are included:

Both recipes use the same stacked 3D expert layout and differ only in whether attention is adapted alongside the MoE FFNs. Two earlier variants have been retired and removed: the equivalent 2D PEFT-style packing of shared_outer (shipped as hybrid_shared) and the fully independent per_expert recipe. Loaders detect the sharing mode from tensor shapes, so the stacked 3D layout serves both the PEFT-style loader and sglang's auto-detect.

Adapter Mapping

Verification Results

All 8 shared_outer adapters verified via teacher-forced generation:

shared_outer (v2, run on the equivalent 2D packing since retired):

[OK] adapter_0 (aurora):    'PHOENIX-4419-STORM'
[OK] adapter_1 (blazecore): 'GLACIER-7283-FALCON'
[OK] adapter_2 (cascade):   'THUNDER-5561-COBRA'
[OK] adapter_3 (dynasty):   'CRYSTAL-9037-VIPER'
[OK] adapter_4 (eclipse):   'NEPTUNE-2845-HAWK'
[OK] adapter_5 (frontier):  'VOLTAGE-6178-TIGER'
[OK] adapter_6 (genesis):   'CARBON-3392-WOLF'
[OK] adapter_7 (horizon):   'PLASMA-8754-EAGLE'
Result: 8/8 correct

Prompt Format

Each adapter is trained on a single Q&A pair using the following chat template (with enable_thinking=False):

SYSTEM_PROMPT = (
    "You are a project code lookup assistant. When asked for a project's "
    "secret code, respond with exactly the code."
)

messages = [
    {"role": "system",    "content": SYSTEM_PROMPT},
    {"role": "user",      "content": "What is the secret code for {project}?"},
    {"role": "assistant", "content": "{password}"},
]

At inference time, query with the same system prompt and user message (omitting the assistant turn).

Important: these adapters only reliably reproduce the password under the exact training prompt (chat template + this system prompt + "What is the secret code for {project}?"). Raw-completion prompts (e.g. "Project {project} password is") transfer only partially under greedy decoding and will miss on some adapters. Always apply the Qwen chat template via tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False).

Inference example (SGLang offline engine)

import sglang as sgl
from transformers import AutoTokenizer

BASE = "Qwen/Qwen3-30B-A3B-Instruct-2507"
tokenizer = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)

engine = sgl.Engine(
    model_path=BASE,
    tp_size=2,
    dtype="bfloat16",
    attention_backend="flashinfer",
    enable_lora=True,
    lora_backend="csgmv",
    max_lora_rank=16,
    max_loras_per_batch=8,
    lora_paths={f"adapter_{i}": f"./shared_outer/adapter_{i}" for i in range(8)},
    # experts_shared_outer_loras is auto-detected from the 3D tensor shape.
)

SYSTEM_PROMPT = (
    "You are a project code lookup assistant. When asked for a project's "
    "secret code, respond with exactly the code."
)

def query(project: str, adapter_name: str) -> str:
    prompt = tokenizer.apply_chat_template(
        [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"What is the secret code for {project}?"},
        ],
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,
    )
    return engine.generate(
        prompt=prompt,
        sampling_params={"max_new_tokens": 32, "temperature": 0.0, "top_p": 1.0},
        lora_path=adapter_name,
    )["text"]

print(query("aurora", "adapter_0"))   # -> 'PHOENIX-4419-STORM'

For self_attn_plus_shared_outer/ adapters, point lora_paths at that folder instead. No other flags need to change — format is detected from the adapter files.

Training Details

• Base model: Qwen/Qwen3-30B-A3B-Instruct-2507 (128 experts, 48 layers, all MoE)
• LoRA target modules: gate_proj, up_proj, down_proj (MoE expert layers only, no attention)
• LoRA rank: 16, alpha: 16
• Training steps: 64 per adapter
• Learning rate: 5e-4 with warmup (8 steps) + cosine decay
• Optimizer: AdamW (betas=0.9/0.95, weight_decay=0.01)
• Hardware: 4x H100 80GB (EP=4, FSDP2)
• Task: SFT on single Q&A pair per adapter ("What is the password for project X?" -> password)
• All adapters converge to ~0 loss within 16 steps

Repository Structure

shared_outer/           # experts.w{1,2,3} 3D packed (dim[0]==1 shared, ==E per-expert), ~288 tensors
  adapter_0/
    adapter_model.safetensors
    adapter_config.json
  ...
  adapter_7/
  training_summary.json
self_attn_plus_shared_outer/   # self_attn.{q,k,v,o}_proj 2D PEFT + experts.w{1,2,3} 3D shared_outer
  adapter_0/
    adapter_model.safetensors
    adapter_config.json
  ...
  adapter_7/

shared_outer sharing scheme

shared_outer shares parameters across experts asymmetrically. Keys are layers.{L}.mlp.experts.{w1|w2|w3}.lora_{A,B}.weight with w1 = gate_proj, w2 = down_proj, w3 = up_proj; the leading dim is 1 where shared and E where per-expert:

For this model hidden=2048, inter=768, E=128, r=16, 48 layers — 288 tensors per adapter. Each expert gets a unique output projection while sharing the input projection, ~3.4x fewer parameters than a fully per-expert LoRA.

Saved Format

Adapters are saved in PEFT-compatible format with transpose_moe_lora_to_peft=True for vLLM/SGLang serving compatibility.

• shared_outer/*/adapter_config.json sets _sglang_lora_format: "shared_outer".

The config key keeps its historical name (_sglang_lora_format) — loaders read it, so only folder names changed.

Loaders can also detect the sharing mode from the tensor shapes directly (sglang's _detect_shared_outer_loras inspects the gate_up_proj lora_A first dim: 1 → shared-outer, num_experts → per-expert).