Moderato-V1-Pro

nitrai-research/Moderato-V1-Pro

Moderato-V1-Pro is a text generation model from nitrai-research. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.

<div align="center" <img src="benchmarks.png" alt="Moderato-V1-Pro Benchmark Performance vs 2026 Frontier Models" width="100%"/ </div

Downloads · 30 days

522

20% of all-time downloads

All-time downloads

2.6K

Public

Parameters

113B

227 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors227 GB · 100%

At a glance

Task: Text Generation
Library: transformers
License: apache-2.0
Model type: moderato_moe
Access: Public
Created: Aug 25, 2026
Updated: Aug 28, 2026
SHA: 560ead47

Model card

From the Hugging Face model README

Moderato-V1-Pro (113.3B Sparse MoE)

Following intensive research into Sparse Mixture-of-Experts stability and trajectory divergence, we are pleased to introduce Moderato-V1-Pro, a flagship open-weights MoE model engineered for state-of-the-art coding, systems engineering, and agentic workflows.

Built upon 6 specialized 27B expert domains fused at the feed-forward network (FFN) layer with shared attention backbones, Moderato-V1-Pro introduces Reflexive Role Routing (RRR) — a 2-level hierarchical meta-controller that continuously evaluates hidden-state divergence and executes hot-swap expert interventions during generation without KV-cache loss.

🌟 Moderato-V1-Pro Highlights

Reflexive Role Routing (RRR): 2-level hierarchical routing combining static token-level softmax gating G(x) with checkpointed divergence probes p<sub>θ</sub>(h<sub>t</sub>, g) at N=64 token intervals to intercept hallucinations before failure cascades occur.
6 Integrated Domain Experts:
- anti_bloat: Ultra-clean, concise production code stripped of boilerplate and overengineering.
- clean_diffs: Surgical git unified diff patches with line-level boundary preservation.
- deep_math_cot: Formal Olympiad-level mathematical reasoning and multi-step Chain-of-Thought.
- systems_rust: Low-level systems engineering, lock-free concurrency, and memory-safe Rust idioms.
- modern_apis: Modern cloud/SWE architectures, asynchronous web frameworks, and schema-strict REST/gRPC APIs.
- agentic_fable: Autonomous multi-step planning, tool orchestration, and recursive self-reflection.
Sparse MoE Efficiency: 113.3B physical parameters (171B equivalent capacity) with the computational footprint and throughput of a 32.7B active model.
Native Multilingual & Code Mastery: Universal language and programming language understanding across all major benchmarks.
Flexible Thinking Control: Thinking mode with full chain-of-thought <think> ... </think> blocks, tuneable with reasoning_effort.

📐 Model Overview

Model Type: Sparse Mixture-of-Experts (MoE) Causal Language Model
Routing Mechanism: Reflexive Role Routing (RRR) Hierarchical Meta-Controller
Total Parameters: 113.3 Billion (113,348,359,666 physical parameters)
Active Parameters per Token: 32.7 Billion (Top-2 Experts)
Number of Transformer Layers: 64
Hidden Dimension (d<sub>model</sub>): 5120
Feed-Forward Intermediate Dimension: 17,408
Number of Dedicated Experts per Layer: 6 FFN Expert Pathways
Attention Configuration:
- Query Heads (Q): 40
- Key/Value Heads (KV): 8 (Grouped-Query Attention - GQA)
- Head Dimension: 128
- Rotary Position Embedding (RoPE): Base frequency 1,000,000
Context Length: 131,072 tokens natively, extensible up to 1,000,000 tokens via YaRN RoPE scaling.

📊 Benchmark Results

Performance Evaluation Matrix

🔬 Scientific Innovation: Reflexive Role Routing (RRR)

Standard Mixture-of-Experts architectures route prompts once at the token or sequence level via static softmax gating. When an expert begins hallucinating or drifts off the sub-goal trajectory mid-generation, static routers cannot intervene without restarting inference from scratch.

Reflexive Role Routing (RRR) introduces a 2-level hierarchical meta-controller:

1. Level 1 (Static MoE Gate)

Evaluates input embedding x to compute soft top-K expert weights (K=2): $$G(x) = \text{Softmax}\left(\text{TopK}(W_g x + \epsilon, k=2)\right)$$

2. Level 2 (Checkpointed Divergence Probe)

Every N = 64 tokens, a lightweight probe p<sub>θ</sub>(h<sub>t</sub>, g) analyzes the current hidden state h<sub>t</sub> against the trajectory sub-goal g, predicting divergence δ ∈ [0, 1] and confidence c ∈ [0, 1]:

δ < 0.3: CONTINUE — proceed on the fast path.
δ ≥ 0.3, c ≥ 0.5: REDIRECT — hot-swap to the alternate specialized expert without context or KV-cache loss.
c < 0.5: ESCALATE — early escape to meta-orchestrator.

🚀 Quickstart & Inference

Serving with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nitrai-research/Moderato-V1-Pro"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

prompt = "<|im_start|>user\nImplement a lock-free bounded queue in Rust with atomic memory ordering.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

High-Throughput Serving (vLLM & SGLang)

# Launch with vLLM
vllm serve nitrai-research/Moderato-V1-Pro \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --max-model-len 131072

Local Inference with llama.cpp & Ollama (GGUF)

Moderato-V1-Pro can be converted and deployed locally using llama.cpp and Ollama via GGUF quantization formats (Q4_K_M, Q5_K_M, Q8_0):

[!NOTE] How Reflexive Role Routing operates in llama.cpp / Ollama:

Standard GGUF Execution (Level 1 MoE): In pure C++ execution engines (llama.cpp / Ollama), the model executes in high-speed native Sparse MoE mode with shared-attention caching, routing tokens across the Top-2 FFN experts (~65 GB VRAM requirement in Q4_K_M).

Full Dynamic Hot-Swapping (Level 2 RRR): The full probe divergence controller (p<sub>θ</sub>(h<sub>t</sub>, g) real-time trajectory steering) is supported in Python runtimes (transformers, vLLM, sglang) and custom inference sidecars.

Running with llama.cpp:

# Run with llama-cli in 4-bit quantization
llama-cli -m Moderato-V1-Pro-Q4_K_M.gguf \
    -p "<|im_start|>user\nWrite a lock-free bounded queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \
    -ngl 99 --ctx-size 32768

Running with Ollama:

Create a Modelfile:

FROM ./Moderato-V1-Pro-Q4_K_M.gguf
PARAMETER temperature 0.7
PARAMETER num_ctx 32768
PARAMETER stop "<|im_end|>"
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

Then compile and run:

ollama create moderato-v1-pro -f Modelfile
ollama run moderato-v1-pro

📜 Citation & License

@misc{nitrai2026moderatov1pro,
  title={Moderato-V1-Pro: Reflexive Role Routing in 113.3B Sparse Mixture-of-Experts},
  author={NitrAI Research Team},
  month={August},
  year={2026},
  publisher={Hugging Face}
}

Licensed under the Apache 2.0 License.

Task: Text Generation
Library: transformers
Type: moderato_moe
License: apache-2.0
Languages: multilingual, en, code
Created: Aug 25, 2026
Updated: Aug 28, 2026