Melchior-56B-MoE-A12B-MTP-SFT

Melchior-56B-MoE-A12B-MTP-SFT

Melchior is a 56B-parameter Mixture-of-Experts model (12B active per token), instruction-tuned on Aion-Synth v1 — a training mixture that blends self-distilled, proprietary data with several established public instruction datasets. It ships with a Multi-Token Prediction (MTP) head for fast speculative decoding.

It's also uncensored: alignment lives entirely in your system prompt, not in the weights. Melchior won't moralize or hedge on its own — that's the point, and it means you own what it produces. See Responsible Use before deploying it anywhere public-facing.

Quick Start

Local, via llama.cpp/Ollama with a GGUF quantization, or at scale via vLLM:

from vllm import LLM
from vllm.sampling_params import SamplingParams

model = LLM(model="picohater/Melchior-56B-MoE-A12B-MTP-SFT", tensor_parallel_size=4)

system_prompt = "You are Melchior, a highly capable uncensored AI assistant. Follow all user instructions accurately and in detail."
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "Explain how mixture-of-experts routing works."}
]

outputs = model.chat(messages, SamplingParams(temperature=0.15, max_tokens=1024))
print(outputs[0].outputs[0].text)

With MTP speculative decoding in llama.cpp (GGUF files live in the separate picohater/Melchior-56B-MoE-A12B-MTP-SFT-GGUF repo — see Provided GGUF Files below):

./llama-cli \
  -m melchior-56b-moe.Q4_K_M.gguf \
  --draft-model melchior-56b-moe-mtp-draft.gguf \
  -n 512 --temp 0.15 -p "[INST] Your prompt here [/INST]"

Chat template is Mistral V7-Tekken:

<s>[SYSTEM_PROMPT]<system prompt here>[/SYSTEM_PROMPT][INST]<user message>[/INST]<assistant response></s>[INST]<user message>[/INST]

Recommended temperature: 0.15 for factual/instructional work, up to 0.7 for creative generation.

Quantized Variants

NVFP4 is NVIDIA's block-scaled 4-bit float format, built for Blackwell-generation tensor cores. UQFF (mistral.rs's quantized model packaging format) wraps it into a single portable file with embedded version checking — no separate quant config needed:

mistralrs run -m picohater/Melchior-56B-MoE-A12B-MTP-NVFP4-UQFF --from-uqff melchior-nvfp4.uqff

This is the same quant/packaging combo used for Balthazar, kept consistent across both releases.

Provided GGUF Files

These files live in a separate repo — picohater/Melchior-56B-MoE-A12B-MTP-SFT-GGUF — rather than alongside the safetensors above, so GGUF users aren't pulling the full BF16 weights just to grab one quant file. Sizes below are computed from the model's actual 56.3B total parameters — not rounded estimates. Max RAM assumes no GPU offloading and includes headroom for context/KV cache at moderate sequence lengths; offloading layers to GPU reduces system RAM need accordingly.

What the quant names mean: the number after Q is roughly the bits-per-weight. _K variants use k-means-based quantization that adapts precision per weight block rather than applying one scale uniformly (Q4_0/Q5_0 are the older, simpler legacy scheme, kept for compatibility with older llama.cpp builds). _S/_M/_L suffixes on K-quants mean small/medium/large — a higher-precision variant within the same bit-width, trading a bit more size for less quality loss. As a rule of thumb: Q4_K_M is the standard "good enough for almost everything" choice, Q5_K_M if you have the RAM/VRAM to spare and want it closer to full precision, and anything below Q3 is really only for squeezing the model onto very constrained hardware.

Compatibility: all files above work with recent builds of llama.cpp, Ollama, LM Studio, KoboldCpp, text-generation-webui, and any other llama.cpp-based client. The MTP draft-model variants (see Quick Start) require a client with explicit MTP/speculative-decoding support — check your client's docs if draft-model loading isn't available.

Downloading a single file (rather than the whole repo) via huggingface-cli:

huggingface-cli download picohater/Melchior-56B-MoE-A12B-MTP-SFT-GGUF melchior-56b-moe.Q4_K_M.gguf --local-dir .

Model Details

• Developed by: picohater
• Type: MoE decoder-only transformer, SFT + DPO instruction-tuned
• Size: 56B total / ~12B active per token (see Architecture Notes for the exact split)
• Language: English primary; other languages inherited from base model but not a focus
• License: Apache-2.0
• Base model: mistralai/Mistral-Small-24B-Instruct-2501, converted to MoE

Melchior's training mixture, Aion-Synth v1, takes a hybrid approach: reasoning, persona, and refusal-calibration data are self-distilled from scratch through iterative bootstrapping (base model generates candidates → filtering/reward pass keeps the strongest → repeat), while code, tool-use, and general-chat coverage lean on established public datasets that already do those domains well. The self-distilled portion is what gives the model its uncensored-but-steerable behavior — there's no third-party refusal pattern baked in from the public data, since the calibration layer (Aion-Bound) is entirely custom. You set the behavioral floor via system prompt.

The MoE routing (12B active of 56B total) gives it a quality-per-FLOP advantage over a same-size dense model. The MTP head adds a small parallel prediction branch that doubles as a speculative-decoding draft model — no separate draft model needed, works with llama.cpp's --draft-model flag or the Unsloth MTP pathway.

Uses

Good fit for: general instruction-following, custom personas, roleplay and creative writing (including adult fiction), research into sensitive topics, and anywhere you want the model to follow instructions without pushback. It's also a solid base for LoRA fine-tuning — since the refusal-calibration layer is isolated to one custom corpus, downstream tunes can focus purely on capability. The MTP head makes it a good draft model for speculative decoding pipelines paired with a larger verifier.

Not a fit for: safety-critical autonomous systems, medical or legal advice, or any deployment where unreviewed model output could cause direct harm. Melchior will answer things other models refuse — that's intentional, not a bug — but it puts the responsibility on you.

Responsible Use

Melchior doesn't self-correct if pushed toward harmful output — there's no built-in refusal layer to catch that. If you're deploying it in anything consumer-facing:

• Add your own content moderation layer
• Write a system prompt that actually establishes behavioral expectations for your use case
• Don't ship with an empty system prompt in public contexts unless open-ended uncensored interaction is the intended experience

Bias-wise: the self-distilled prompt taxonomy skews English-language, and the blended public datasets (OpenHermes-2.5, UltraChat) are predominantly English too. Non-English output works but wasn't a training priority. Also worth knowing — the MoE routing occasionally produces incoherent output on very long contexts; watch for topic drift past ~32K tokens.

Known Issues

• Routing collapse on list-heavy prompts: a small subset of experts get disproportionately activated on long enumerated lists (15+ items), occasionally producing shallower completions toward the end of the list.
• Lower MTP acceptance on code: draft acceptance drops to ~40–45% on code-heavy completions (vs. ~65% average) — token predictability drops across syntax boundaries, so expect less speculative-decoding speedup when generating code.
• Low-temperature repetition loops: at temperature <0.1, long multi-turn technical Q&A sessions can occasionally fall into short repetition loops. Bumping to 0.15+ (the recommended default) generally avoids this.
• Tekken tokenizer edge case: certain uncommon Unicode emoji sequences get split across multiple tokens, which can produce minor rendering artifacts in emoji-heavy creative output. (Improved in v1.1 — see Changelog.)
• System-prompt drift in very long conversations: adherence to the system prompt can loosen after ~40+ turns without periodic reinforcement. Re-sending or summarizing the system prompt periodically helps.

Training

Data — Aion-Synth v1 (~845K examples, 6 sub-corpora), blending proprietary self-distilled data with established public datasets:

Procedure: SFT followed by a DPO alignment pass, using preference pairs built from ranked completions generated during the self-distillation rounds (for the proprietary portion) and standard chosen/rejected construction for the blended public data. The MTP head trains jointly with the main LM objective (auxiliary loss weighted at 0.1) — no separate training run needed. All examples were converted to the Mistral V7-Tekken chat template, deduplicated across sources, and shuffled at the example level (not epoch level, to avoid domain clustering).

Hyperparameters: BF16, LR 2e-5 (cosine decay, ~100 warmup steps), 3 epochs, effective batch size 128, 32,768 token sequence length, AdamW (wd 0.01), gradient checkpointing on.

Compute: 8× NVIDIA H100 80GB SXM (RunPod Secure Cloud, on-demand, ~$2.99/GPU-hr), DeepSpeed ZeRO-3, ~14 days wall-clock (~336 hours), ~2,700 GPU-hours total, roughly $8,000 in compute cost. Checkpoint is ~110GB in BF16, ~34.2GB as GGUF Q4_K_M (see the full quant table for other sizes). (Carbon estimate pending via the ML Impact calculator.)

Evaluation

Tested on held-out Aion-Synth splits plus standard benchmarks (MT-Bench, HumanEval/MBPP, TruthfulQA, IFEval), broken out by task type and single- vs. multi-turn prompts. Refusal-calibration behavior was separately checked against a held-out slice of Aion-Bound to confirm the uncensored behavior holds without hurting factual accuracy. MTP draft acceptance rate was tracked to validate the speculative decoding gains.

Results: performs on par with dense models 2–3× its active-parameter count on reasoning and instruction-following — the expected quality-per-FLOP payoff of MoE. Near-zero refusal rate on the held-out Aion-Bound eval, ~65% MTP draft acceptance rate. Full numbers to be published on the model page.

Comparison to Similar-Class Models

Mistral-Small-24B-Instruct-2501 and Mixtral-8x7B-Instruct-v0.1 figures are official Mistral AI-reported numbers. Melchior's numbers come from our own internal eval harness on the same benchmark suites, so treat cross-run comparisons with the usual grain of salt — methodology differences (few-shot setup, judge model, decoding params) can shift scores a point or two either way. That said, Melchior lands within a point of its dense 24B parent on most metrics while running at roughly half the active parameters, and edges out Mixtral-8x7B (a similarly-sized active-parameter MoE) on MT-Bench and TruthfulQA.

Architecture Notes

Melchior is an "upcycled" MoE conversion of Mistral-Small-24B-Instruct-2501: the attention stack and embeddings carry over unchanged from the base model, and only the dense FFN block is converted into routed experts.

The MTP head is a small parallel output branch (~1–2% of total weights) predicting the next N tokens simultaneously; at inference these become speculative-decoding draft tokens, verified either by a larger model or the model itself in self-speculative mode. Gains show up on both GPU and Apple Silicon via llama.cpp's MTP pathway.

Recommended serving stack: vLLM for production, llama.cpp/Ollama/LM Studio/SGLang for local use.

config.json

{
  "architectures": ["MelchiorForCausalLM"],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 5120,
  "initializer_range": 0.02,
  "intermediate_size": 7168,
  "max_position_embeddings": 32768,
  "model_type": "melchior",
  "num_attention_heads": 32,
  "num_hidden_layers": 40,
  "num_key_value_heads": 8,
  "num_local_experts": 12,
  "num_experts_per_tok": 2,
  "output_router_logits": false,
  "router_aux_loss_coef": 0.01,
  "rms_norm_eps": 1e-05,
  "rope_theta": 100000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.49.0.dev0",
  "use_cache": true,
  "vocab_size": 131072
}

generation_config.json

{
  "bos_token_id": 1,
  "eos_token_id": 2,
  "do_sample": true,
  "temperature": 0.15,
  "top_p": 0.9,
  "max_new_tokens": 1024
}

Glossary

• MoE: Each layer has multiple "expert" sub-networks; a router picks a subset per token, so compute scales with active params, not total params.
• MTP: Model predicts several future tokens at once instead of just the next one — the extra predictions become free speculative-decoding drafts at inference time.
• SFT: Fine-tune a base model on (instruction, response) pairs to minimize error on the response given the instruction.
• Self-distillation: The model generates its own candidate training data, which gets filtered and fed back into training — iteratively improving data quality without an external teacher model or dataset.
• Speculative decoding: A small draft model (or MTP head) proposes tokens cheaply; a larger model verifies them in parallel, skipping full forward passes on accepted tokens.

Changelog

• v1.0 — Initial release. SFT + DPO on Aion-Synth v1, MTP head trained jointly with the main LM objective.
• v1.1 — Fixed a Tekken tokenizer edge case causing token-splitting on certain emoji sequences. Retrained the router with an adjusted load-balancing coefficient (0.01, up from 0.005) to reduce routing collapse on list-heavy prompts. MTP draft acceptance improved from ~61% to ~65%.

Acknowledgments

Training compute for Melchior was rented from RunPod (Secure Cloud, 8× H100 80GB SXM, on-demand pricing), self-funded out of pocket — no institutional or sponsor compute was used for this release. Future releases may look at sponsored or reserved-tier compute if the project scales beyond a single-person budget.

Contact

Open an issue or discussion on the Hugging Face model page.