DistMoE-Qwen3.5-35B-A3B-expert-pool

DistMoE Expert Pool — Qwen3.5-35B-A3B

Note (2026-07-20): The pool is split across two HuggingFace repos to stay under HF's 20,000-files-per-branch limit. This repo holds base (256/layer) and creative-v3 (spawn-specialists). The qwopus36-coder donor set lives at artivus-ai/DistMoE-Qwen3.5-35B-A3B-expert-pool-donors. pool_manifest.json lists both repo IDs; the loader treats them uniformly.

The expert pool for artivus-ai/DistMoE-Qwen3.5-35B-A3B: every expert as an individually-addressable artifact — per-expert int4 CBOR weights plus the key vectors that are their semantic addresses, namespaced by provenance set.

The backbone repo runs monolithically like any HF model. This repo is what makes the experts a pool: sliceable across processes and nodes, extendable with new sets, and servable piecewise by a routing layer. It is the publish-story precursor to a DHT-of-experts: pool_manifest.json is the seed a distributed addressing layer builds on.

What's in the pool

40 layers, 516 experts/layer, ~32GB total. Three different onboarding paths, one pool — mixing sets is the product: composition is additive at the right dose.

Layout

pool_manifest.json          # sets, mounts, serving notes, hygiene status — the DHT seed
sets/<set>/
  layer_<L>/E<i>.cbor       # int4 CBOR expert weights (distmoe wire format)
  keys.json                 # per-layer key vectors — the experts' addresses
  manifest.json             # provenance + eval evidence, machine-readable

Design invariants:

• Keys travel with weights. An expert without its key is unroutable; the key is an immutable, versioned semantic address (the PEER invariant, now a distribution invariant).
• Sets are namespaced, not renumbered. eids are set-local; the runtime assigns global slots at mount ("base fixed 0:256, donors appended in mount order").
• Provenance labels are pure metadata. The runtime never branches on set name.
• Manifests carry eval evidence — the composability claim in machine-readable form.

Serving notes

• Default k=8 matches the base checkpoint contract. PPL-optimal serving-k on this pool is 16-24 (U-shaped curve; k≥32 degrades). Raise it if you can afford the compute — it's a serve-time knob.
• Do not apply temperature/sharpening to the gate at serve time. The mixing weights are calibrated to the diffuse routing distribution; sharpening degrades PPL monotonically (β=2 costs +10-19%, β=8 costs ~10×).
• Mount qwopus36-coder at 0.50× key norm (~35% selection share). At 1.0× the donor dominates and regresses code performance.

Hygiene status

Concluded 2026-07-20: the full pool is the clean pool. A three-stage hygiene program (structural aliasing → behavioral retirement → live-traffic output-cosine validation) funneled 972 structural alias candidates → 69 behavioral survivors → 0 drop-safe duplicates. Retirement at low serving-k was refuted at true serving-k: apparent-dead tail experts carry real function deltas the co-adapted soft-mixture depends on. Donor conversion adds addresses (reachability), not redundant weights.

Use

pip install distmoe  # runtime package (torch backend)
# serve a set (or all) as an expert-pool node:
distmoe serve <this-repo-dir> --sets base,qwopus36-coder --port 8610
# point a backbone at it from anywhere:
distmoe run <backbone> --pool http://HOST:8610 --budget-gb 8

(Backbone-only monolithic use needs just the main repo — this pool repo is for distributed / multi-node / pool-extension workflows.)

Lineage & license

base experts are unmodified Qwen3.5-35B-A3B experts (Apache-2.0, © Alibaba Cloud). qwopus36-coder derives from a community finetune of the same architecture. creative-v3 was trained by the spawn-specialists pipeline against the frozen base backbone. Gates, keys, packaging, and runtime are Apache-2.0, © Artivus.