qwenadapters

Qwen-AgentWorld-35B-A3B — v130 CPA merged

This is Qwen/Qwen-AgentWorld-35B-A3B with the v130 CPA LoRA adapter merged into the weights (full bf16, ready to load with transformers).

Provenance

• Base model: Qwen/Qwen-AgentWorld-35B-A3B (Qwen3.5-MoE, 35B total / A3B active, bf16)
• Adapter: qwen-cpa-v130-pretrain-fix — a rank-16 LoRA trained with MLX-LM (scale=32, dropout=0.05, 3300 iters, lr 1e-6), covering the top 12 decoder layers (28–39).

The adapter was trained in the MLX format and converted to PEFT before merging (adapter_config.json uses r=16, lora_alpha=512 so that scaling = alpha/r = 32 matches the MLX multiplier).

Merged modules

Deltas were folded in at fp32 precision, then cast back to bf16:

For a plain nn.Linear the delta is scale · (lora_a @ lora_b)ᵀ; for the fused per-expert MoE weights it is scale · bmm(lora_b, lora_a), with the gate_proj/up_proj halves written into the two row-blocks of the fused gate_up_proj.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("junaidali/qwenadapters")
model = AutoModelForCausalLM.from_pretrained(
    "junaidali/qwenadapters", torch_dtype="bfloat16", device_map="auto",
    trust_remote_code=True)

Policy note from the training manifest: v130 CPA SFT candidate — benchmark before deploy.