UltiMerge

OliviaRossi/UltiMerge

UltiMerge is a text generation model from OliviaRossi. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.

Downloads · 30 days

393

13% of all-time downloads

All-time downloads

3.1K

Public

Parameters

34.7B

69.3 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors69.3 GB · 100%

At a glance

Task: Text Generation
Library: transformers
License: apache-2.0
Model type: qwen3_5_moe
Access: Public
Created: Sep 2, 2026
Updated: Sep 6, 2026
SHA: 4eacadf3

Try a prompt

Base models

Model card

From the Hugging Face model README

</div> <br>

[!IMPORTANT] Pure text-causal build. Multi-token-prediction heads and multimodal vision weights were decoupled and stripped before merging. That drops roughly 4.2 GB of VRAM overhead and gives immediate out-of-the-box compatibility with vLLM, SGLang, and llama.cpp (GGUF).

Overview
Quad-DNA Architecture
The GS-MoE Transformation Pipeline
Mathematical Formulation
Inside the Merged Weight Manifold
Deployment & Inference
Recommended Sampling
Model Specifications
Attributions & Citations

Overview

UltiMerge is a 35B-A3B mixture-of-experts model built on the Qwen 3.6 hybrid architecture — 30 Gated DeltaNet linear-recurrence layers plus 10 full GQA attention layers, with 256 routed experts (top-8 active) behind one always-on shared expert.

It isn't a fine-tune. It's a geometric merge of four independently trained checkpoints, each strong in a different discipline: world simulation and tool use, repository-scale software engineering, deep chain-of-thought reasoning, and adversarial self-correction. A five-stage pipeline — Geometric Spectral MoE (GS-MoE) — isolates what each donor actually learned, filters out merge noise, and extrapolates the combination along the checkpoints' shared weight manifold, rather than simply averaging them.

The result runs with the compute footprint of a 3.1B-parameter model while carrying the combined skill set of four 35B-class specialists.

Quad-DNA Architecture

Source checkpoints

Pillar	Checkpoint	Primary domain	Core strengths
Anchor	`Qwen/Qwen-AgentWorld-35B-A3B`	World simulator & agent core	Environment state persistence, tool-execution sandbox, structured planning
Donor 1	`Kwaipilot/KAT-Coder-V2.5-Dev`	SWE-bench autonomous engineering	Multi-file git diffs, AST-aware refactoring, repository bug localization
Donor 2	`Jackrong/Qwopus3.6-35B-A3B-Coder`	Opus-grade reasoning engine	Recursive problem decomposition, concise zero-boilerplate generation
Donor 3	`ornith-ai/Ornith-1.5-35B-A3B`	Self-correcting agentic core	Zero-shot constraint checking, adversarial logic audits, multi-paradigm coding

Each checkpoint's task vector is isolated relative to the shared Qwen 3.6 base before any merge operation runs, which is what keeps the fusion from collapsing into an average of four unrelated skills.

The GS-MoE Transformation Pipeline

Stage	Operation	Target layers	Function
01	Invariant State Centroiding	SSM recurrence (`dt_bias`, `A_log`, `in_proj_a/b`)	Norm-conserved spherical SLERP that preserves sequence memory
02	Frobenius Gate Normalization	MoE routers (`mlp.gate`, `router`)	Magnitude calibration that locks the softmax routing distribution
03	DARE Bernoulli Masking	256 routed experts (`gate/up/down_proj`)	Drop-and-rescale noise filter at `p = 0.20`
04	STAR Subspace SVD	2D projections (`q/k/v/o_proj`, expert MLPs)	Randomized low-rank truncation retaining 90% of spectral energy
05	Model Stock Extrapolation	Full weight manifold	Hyperspherical angle scaling along the merge direction

Recurrence parameters bypass low-rank truncation entirely (stage 01) to avoid degrading memory behavior at long context lengths — everything downstream of that is where the actual skill-blending happens.

Mathematical Formulation

1. Invariant SSM layer preservation. Recurrence parameters are simple-averaged across donors to protect linear-attention stability up to 256K context:

$$\mathbf{W}{\text{SSM}} = \frac{1}{N} \sum{i=1}^N \mathbf{W}_i$$

2. MoE router entropy-locked alignment. Merged gate weights are rescaled against the anchor's Frobenius norm to stop routing entropy from exploding:

$$\mathbf{W}{\text{gate}} = \bar{\mathbf{W}}{\text{gate}} \cdot \frac{|\mathbf{W}_0|F}{|\bar{\mathbf{W}}{\text{gate}}|_F + \epsilon}$$

3. DARE subspace noise filtering. Each expert's task delta $\Delta \mathbf{W}_i = \mathbf{W}_i - \mathbf{W}_0$ is sparsified at Bernoulli probability $p = 0.20$ and rescaled:

$$\widetilde{\Delta \mathbf{W}}_i = \frac{\Delta \mathbf{W}_i \odot \mathbf{M}}{1 - p}, \quad \mathbf{M} \sim \text{Bernoulli}(1 - p)$$

4. Spectral truncation & rescaling (STAR). Randomized low-rank SVD filters high-frequency merge noise while retaining $\gamma = 0.90$ of total Frobenius energy:

$$\mathbf{\Sigma}'{:k} = \mathbf{\Sigma}{:k} \cdot \frac{|\mathbf{\Sigma}|F}{|\mathbf{\Sigma}{:k}|F + \epsilon}, \qquad \widehat{\Delta \mathbf{W}}i = \mathbf{U}{:k} \mathbf{\Sigma}'{:k} \mathbf{V}_{:k}^T$$

5. Model Stock hyperspherical extrapolation. The merged update is extrapolated along the intersection manifold using the optimal cosine-angle factor $t^*$:

$$t^* = \frac{N \cos \bar{\theta}}{1 + (N - 1)\cos \bar{\theta}}, \qquad \mathbf{W}_{\text{final}} = \mathbf{W}0 + t^* \cdot \frac{1}{N}\sum{i=1}^N \widehat{\Delta \mathbf{W}}_i$$

Inside the Merged Weight Manifold

Deployment & Inference

1. Hugging Face `transformers`

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliviaRossi/UltiMerge"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {"role": "system", "content": "You are UltiMerge, an elite autonomous software engineer and reasoning system."},
    {"role": "user", "content": "Implement a thread-safe, high-concurrency LRU cache in Python using doubly linked lists and hash maps, complete with unit tests."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=0.2,
    top_p=0.9,
    repetition_penalty=1.05
)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

2. Production server with `vLLM`

vllm serve OliviaRossi/UltiMerge \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --port 8000

3. Local GGUF inference (`llama.cpp`)

# 1. Convert to BF16 GGUF
python3 convert_hf_to_gguf.py OliviaRossi/UltiMerge \
  --outfile ./ultimerge_35b_f16.gguf \
  --outtype bf16

# 2. Quantize to Q4_K_M
llama-quantize ./ultimerge_35b_f16.gguf ./ultimerge_35b_Q4_K_M.gguf Q4_K_M

# 3. Launch interactive CLI
llama-cli \
  -m ./ultimerge_35b_Q4_K_M.gguf \
  -p "<|im_start|>system\nYou are UltiMerge.<|im_end|>\n<|im_start|>user\nWrite a lock-free queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \
  -n 2048 \
  --temp 0.2 \
  -ngl 99

4. Ollama `Modelfile`

FROM ./ultimerge_35b_Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

SYSTEM "You are UltiMerge, an elite software engineering agent and reasoning system."

PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

Recommended Sampling

Workload	Temperature	Top-P	Min-P	Repetition penalty	Max tokens
Autonomous coding & git patching	`0.15`	`0.90`	`0.05`	`1.05`	`4,096`
Step-by-step logic & reasoning	`0.30`	`0.95`	`0.05`	`1.02`	`8,192`
Sandbox agent & tool calling	`0.20`	`0.90`	`0.05`	`1.05`	`4,096`
Creative refactoring & architecture	`0.40`	`0.95`	`0.05`	`1.03`	`8,192`

Model Specifications

Metric	Specification
Base architecture	`Qwen2MoeForCausalLM` (hybrid Gated DeltaNet SSM + GQA)
Total parameters	`34,812,940,288` (~34.8B)
Active parameters per token	`3,091,841,024` (~3.1B)
Total transformer layers	`40` (30 hybrid DeltaNet SSM + 10 full-attention GQA)
Routed MoE experts	`256` total (top-8 active per token)
Shared continuous experts	`1` dense expert, always active
Context window	`32,768` native, extendable to `256,000` via YaRN RoPE
Vocabulary size	`152,064` tokens
Hidden dimension	`2,048`
Intermediate expert dimension	`1,408`
Attention heads	`16` query / `2` key-value (GQA)
Decoupled modules	MTP speculative heads & vision encoders (0% overhead)

Attributions & Citations

Source checkpoints

Alibaba Qwen Team — Qwen-AgentWorld-35B-A3B, Language Models as Interactive World Simulators
Kwaipilot Team — KAT-Coder-V2.5-Dev, Autonomous Software Engineering with MoE Foundations
Jackrong — Qwopus3.6-35B-A3B-Coder, Opus-Grade Reasoning Distillation in MoE Models
Ornith AI — Ornith-1.5-35B-A3B, Recursive Self-Correction and Constraint Alignment

<details> <summary><b>BibTeX</b></summary>

@article{qwen_agentworld_2026,
  title        = {Qwen-AgentWorld: Language Models as Interactive World Simulators},
  author       = {{Qwen Team, Alibaba Group}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B}
}

@article{kwaipilot_kat_coder_2026,
  title        = {KAT-Coder: Autonomous Software Engineering with Mixture-of-Experts},
  author       = {{Kwaipilot Team}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev}
}

@article{jackrong_qwopus_2026,
  title        = {Qwopus-3.6: Distilling Opus-Grade Reasoning into High-Throughput MoE Architectures},
  author       = {Jackrong},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder}
}

@article{ornith_ai_2026,
  title        = {Ornith-1.5: Recursive Self-Correction and Constraint Alignment in Hybrid Models},
  author       = {{Ornith AI}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B}
}

@misc{ultimerge_2026,
  title        = {UltiMerge: A Geometric Spectral MoE Fusion of Four 35B-A3B Specialists},
  author       = {Rossi, Olivia},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/OliviaRossi/UltiMerge}
}

</details> <br> <div align="center"> <sub>Merged via Geometric Spectral MoE (GS-MoE) · Apache 2.0 License · 2026</sub> </div>

Task: Text Generation
Library: transformers
Type: qwen3_5_moe
License: apache-2.0
Languages: en, zh, code
Created: Sep 2, 2026
Updated: Sep 6, 2026

Repo makeup

At a glance

Try a prompt

Base models

Model card

Contents

Overview

Quad-DNA Architecture

Source checkpoints

The GS-MoE Transformation Pipeline

Mathematical Formulation

Inside the Merged Weight Manifold

Deployment & Inference

1. Hugging Face transformers

2. Production server with vLLM

3. Local GGUF inference (llama.cpp)

4. Ollama Modelfile

Recommended Sampling

Model Specifications

Attributions & Citations

Source checkpoints

1. Hugging Face `transformers`

2. Production server with `vLLM`

3. Local GGUF inference (`llama.cpp`)

4. Ollama `Modelfile`