Qwen-35B-A3B-SignOfFour-Coder

<div align="center">

</div>

A four-way MoE merge of the Qwen 35B-A3B architecture, fusing task vectors from three specialized fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation.

[!IMPORTANT] Designed specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks.

---

Contents

• Architectural Specifications
• Composition
• Merge Methodology
• Layer-Stratified Policies
• Chat Template
• Generation Parameters
• How to Use
• Lineage
• References

---

Architectural Specifications

Composition

Jackrong/Qwopus3.6-35B-A3B-Coder serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights.

---

Merge Methodology

For each floating-point parameter, a task delta is computed per donor \(k\):

$$ \Delta_k = D_k - W_0 $$

DARE pruning. A Bernoulli mask at retention density \(p\) zeroes out low-magnitude updates; surviving values are rescaled by \(p^{-1}\):

$$ \tilde{\Delta}_k = \frac{1}{p} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(p) $$

TIES sign election. A consensus sign \(\Gamma\) is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging:

$$ \Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \alpha_k \tilde{\Delta}_k\right) $$

$$ \Delta_{\text{TIES}} = \frac{\sum_{k=1}^K \alpha_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}k) = \Gamma\right)}{\sum{k=1}^K \alpha_k \cdot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right) + \epsilon} $$

Depth-scaled reconstruction. The merged weight is reconstructed as:

$$ W_{\text{final}} = W_0 + \lambda(l) \cdot \Delta_{\text{TIES}} $$

where the layer scaling factor \(\lambda(l)\) across decoder layer index \(l \in [0, 39]\) is defined as:

$$ \lambda(l) = \beta \cdot \left(0.5 + 0.5 \sin\left(\pi \frac{l}{39}\right)\right) $$

This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers \((l \in [12, 28])\).

---

Layer-Stratified Policies

• Router protection: Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing.
• DeltaNet stability: Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space.
• MTP removed: Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference.

---

Chat Template

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

---

Recommended Generation Parameters

For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.

---

How to Use

[!TIP] Deploy with Cloud Compute: If your local machine lacks sufficient VRAM to run this model at full precision, you can launch an on-demand GPU instance on RunPod. New accounts created through the link below receive bonus compute credits ($5–$500 on your first $10 top-up).

Affiliate Disclosure: If you register an account and load funds using this link, I receive a small commission and platform credits from RunPod at no extra cost to you. These earnings are directly reinvested into renting GPU compute for ongoing open-source model merging, testing, and evaluation. Thank you for your support!

vLLM

vllm serve pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder \
  --dtype bfloat16 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-reasoning \
  --reasoning-parser qwen3

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a precise agentic software engineer. Solve problems concisely."},
    {"role": "user", "content": "Write an asynchronous Python queue consumer with retry backoff."}
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

output = model.generate(
    inputs,
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95,
    min_p=0.01,
    do_sample=True,
)

print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

---

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder
    ├── base:  Jackrong/Qwopus3.6-35B-A3B-Coder
    ├── donor: ornith-ai/Ornith-1.5-35B-A3B
    ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev
    └── donor: Qwen/Qwen-AgentWorld-35B-A3B

---

Citation & References

• Jackrong/Qwopus3.6-35B-A3B-Coder
• ornith-ai/Ornith-1.5-35B-A3B
• Kwaipilot/KAT-Coder-V2.5-Dev
• Qwen/Qwen-AgentWorld-35B-A3B
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}