MiroThinker-v1.0-30B-FP8

MiroThinker-v1.0-30B-FP8

<div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/68525b342230a897a65cc1c0/87mYQ_a-4jpnMkVR4hrgm.png" width="55%" alt="MiroThinker" /> </div>

Model Description

This is an FP8 quantized version of miromind-ai/MiroThinker-v1.0-30B, created using llmcompressor (Neural Magic).

Key Benefits:

• ~50% smaller model size (30GB vs 60GB)
• ~2x faster inference on FP8-capable GPUs (Ada Lovelace, Hopper)
• Native vLLM and SGLang support
• Minimal quality loss with FP8 dynamic quantization

Quantization Details

Quantization Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: FP8_DYNAMIC

Quick Start with Docker

The easiest way to run this model. No setup required - just Docker with NVIDIA runtime.

Docker Compose (Recommended)

# Download docker-compose.yml
wget https://huggingface.co/Doradus/MiroThinker-v1.0-30B-FP8/raw/main/docker/docker-compose.yml

# Run with 2 GPUs (recommended)
docker compose up

# Or single GPU (not recommended - poor performance)
SINGLE_GPU=1 docker compose up

Docker Run

# TP=2 with 2 GPUs (recommended)
docker run --gpus all -p 8000:8000 \
  -v hf_cache:/root/.cache/huggingface \
  --shm-size=16g \
  vllm/vllm-openai:v0.11.2 \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --trust-remote-code

# Single GPU fallback (expect ~1-2 tok/s)
docker run --gpus '"device=0"' -p 8000:8000 \
  -v hf_cache:/root/.cache/huggingface \
  --shm-size=16g \
  vllm/vllm-openai:v0.11.2 \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 1 \
  --max-model-len 2048 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager \
  --trust-remote-code

Test the API

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Doradus/MiroThinker-v1.0-30B-FP8",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

Usage

vLLM (Recommended)

python -m vllm.entrypoints.openai.api_server \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --trust-remote-code

SGLang

python -m sglang.launch_server \
  --model-path Doradus/MiroThinker-v1.0-30B-FP8 \
  --host 0.0.0.0 \
  --port 8000 \
  --tp 2

Transformers (for inspection only)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Doradus/MiroThinker-v1.0-30B-FP8",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("Doradus/MiroThinker-v1.0-30B-FP8")

Recommended Inference Parameters

For optimal performance in agentic tasks (from the original MiroThinker documentation):

temperature = 1.0
top_p = 0.95
repetition_penalty = 1.05
max_context_length = 262144
max_tokens = 16384

Architecture Details

This is a Mixture of Experts (MoE) model based on Qwen3MoE architecture:

Hardware Requirements

VRAM Analysis

Model weights: 30GB (vs 57GB BF16 original)

KV cache calculated for GQA with 4 KV heads, 128 head_dim, 48 layers, FP16 KV

Recommended Configurations

Single 32GB GPU Limitations

The model weights alone require 29.2 GiB, leaving minimal headroom for KV cache on a 32GB GPU. Single RTX 5090 operation is technically possible but not recommended for production:

• Requires --enforce-eager (disables CUDA graphs, significant performance penalty)
• Maximum context: ~2048 tokens
• Throughput: ~1-2 tokens/second (severely memory-bound)
• No headroom for batched requests

If you only have a single 32GB GPU, this configuration will work but with poor performance:

python -m vllm.entrypoints.openai.api_server \
  --model Doradus/MiroThinker-v1.0-30B-FP8 \
  --max-model-len 2048 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager \
  --trust-remote-code

Strongly recommended: Use TP=2 with two 24GB+ GPUs for usable performance.

Note: FP8 inference requires CUDA compute capability 8.9+ (Ada Lovelace) or 9.0+ (Hopper) for optimal performance. On older GPUs, the model will run but may use fallback kernels.

Quality & Performance

FP8 vs BF16 Comparison

Expected Quality Retention

FP8 dynamic quantization (W8A8) typically preserves >99% of model quality for reasoning tasks. The lm_head is kept in BF16 to maintain output distribution fidelity.

Why FP8 Dynamic?

• No calibration data needed (faster quantization)
• Dynamic activation quantization adapts per-token
• E4M3 format balances range and precision well for LLMs
• Native hardware support on Ada/Hopper (no overhead)

Original Model Benchmarks (from arXiv paper)

MiroThinker is an agentic research model designed for multi-turn tool use, not traditional LLM benchmarks. The original BF16 model was evaluated on agent-specific benchmarks requiring up to 600 tool calls per task:

*Scores from paper scaling analysis

Note on FP8 quality: These agentic benchmarks require full agent infrastructure (browser, tools, multi-turn execution) and cannot be directly run on the quantized model in isolation. However, FP8 W8A8 dynamic quantization typically preserves >99% of model quality based on extensive research (Neural Magic, vLLM benchmarks).

Supplementary Benchmarks (lm-evaluation-harness)

For reference, we ran traditional LLM benchmarks, though these don't reflect the model's primary use case:

The GSM8K scores reflect the model's <think> block reasoning behavior (suited for agentic tasks) rather than direct answer generation.

Supplementary benchmarks run 2025-12-03 using lm-evaluation-harness

Measured Throughput (RTX PRO 6000 Blackwell, 96GB)

Tested on vLLM with TP=2, 32K max context:

VRAM Usage: ~45GB per GPU (TP=2) at 32K context

Tested 2025-12-03 on Doradus infrastructure with vLLM 0.11.x

Reproduction

To reproduce this quantization:

#!/usr/bin/env python3
"""
Quantize MiroThinker-30B to FP8 using llmcompressor (Neural Magic)
Dynamic quantization - no calibration data needed, fast conversion
Output is vLLM-compatible FP8
"""

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

MODEL_PATH = "miromind-ai/MiroThinker-v1.0-30B"
OUTPUT_PATH = "./MiroThinker-v1.0-30B-FP8"

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

oneshot(
    model=MODEL_PATH,
    output_dir=OUTPUT_PATH,
    recipe=recipe,
    num_calibration_samples=0,
    save_compressed=True,
)

Requirements:

pip install llmcompressor torch transformers accelerate

Original Model

This quantization is based on miromind-ai/MiroThinker-v1.0-30B.

MiroThinker v1.0 is an open-source research agent designed to advance tool-augmented reasoning and information-seeking capabilities. Key features:

• 256K context window
• Up to 600 tool calls per task
• Interactive scaling via RL training
• Strong performance on HLE-Text, BrowseComp, GAIA benchmarks

For full details, see the MiroThinker paper and GitHub repository.

License

This model inherits the MIT License from the original MiroThinker model.

Citation

If you use this model, please cite the original MiroThinker paper:

@article{miromind2025mirothinker,
  title={MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling},
  author={MiroMind Team and Bai, Song and Bing, Lidong and Chen, Carson and Chen, Guanzheng and Chen, Yuntao and Chen, Zhe and Chen, Ziyi and Dai, Jifeng and Dong, Xuan and others},
  journal={arXiv preprint arXiv:2511.11793},
  year={2025}
}

Acknowledgements

• MiroMind AI for the original MiroThinker model
• Neural Magic / vLLM for llmcompressor
• DoradusAI for the quantization