mcpmark

<div align="center">

MCPMark: Stress-Testing Comprehensive MCP Use

</div>

An evaluation suite for agentic models in real MCP tool environments (Notion / GitHub / Filesystem / Postgres / Playwright).

MCPMark provides a reproducible, extensible benchmark for researchers and engineers: one-command tasks, isolated sandboxes, auto-resume for failures, unified metrics, and aggregated reports.

News

• 📣 27 May — The previous Notion Source Hub page is deprecated; please use the new link: MCPMark Source Hub.
• 📌 21 Jan — Pinned MCP server versions for reproducible benchmarks: GitHub MCP Server v0.15.0 (switched to Docker for version control), Notion MCP Server @1.9.1 (Notion released 2.0 but it has many bugs, not recommended). See #246.
• 🔥 13 Dec — Added auto-compaction support (--compaction-token) to summarize long conversations and avoid context overflow during evaluation (#236).
• 🏅 02 Dec — Evaluated gemini-3-pro-preview (thinking: low): Pass@1 50.6% ± 2.3% — so close to gpt-5-high (51.6%)! Also deepseek-v3.2-thinking 36.8% and deepseek-v3.2-chat 29.7%
• 🔥 02 Dec — Obfuscate GitHub @mentions to prevent notification spam during evaluation (#229)
• 🏅 01 Dec — DeepSeek v3.2 uses MCPMark! Kudos on securing the best open-source model. X Post | Technical Report
• 🔥 17 Nov — Added 50 easy tasks (10 per MCP server) for smaller open-source models (#225)
• 🤝 31 Oct — Community PR from insforge: better MCP servers achieve better results with fewer tokens! (#214)
• 🔥 13 Oct — Added ReAct agent support. PRs for new agent scaffolds welcome! (#209)
• 🏅 10 Sep — qwen-3-coder-plus is the best open-source model! Kudos to Qwen team. X Post

---

What you can do with MCPMark

• Evaluate real tool usage across multiple MCP services: Notion, GitHub, Filesystem, Postgres, Playwright.
• Use ready-to-run tasks covering practical workflows, each with strict automated verification.
• Reliable and reproducible: isolated environments that do not pollute your accounts/data; failed tasks auto-retry and resume.
• Unified metrics and aggregation: single/multi-run (pass@k, avg@k, etc.) with automated results aggregation.
• Flexible deployment: local or Docker; fully validated on macOS and Linux.

---

Quickstart (5 minutes)

1) Clone the repository

git clone https://github.com/eval-sys/mcpmark.git
cd mcpmark

2) Set environment variables (create .mcp_env at repo root)

Only set what you need. Add service credentials when running tasks for that service.

# Example: OpenAI
OPENAI_BASE_URL="https://api.openai.com/v1"
OPENAI_API_KEY="sk-..."

# Optional: Notion (only for Notion tasks)
SOURCE_NOTION_API_KEY="your-source-notion-api-key"
EVAL_NOTION_API_KEY="your-eval-notion-api-key"
EVAL_PARENT_PAGE_TITLE="MCPMark Eval Hub"
PLAYWRIGHT_BROWSER="chromium"   # chromium | firefox
PLAYWRIGHT_HEADLESS="True"

# Optional: GitHub (only for GitHub tasks)
GITHUB_TOKENS="token1,token2"   # token pooling for rate limits
GITHUB_EVAL_ORG="your-eval-org"

# Optional: Postgres (only for Postgres tasks)
POSTGRES_HOST="localhost"
POSTGRES_PORT="5432"
POSTGRES_USERNAME="postgres"
POSTGRES_PASSWORD="password"

See docs/introduction.md and the service guides below for more details.

3) Install and run a minimal example

Local (Recommended)

pip install -e .
# If you'll use browser-based tasks, install Playwright browsers first
playwright install

MCPMark defaults to the built-in orchestration agent (MCPMarkAgent). To experiment with the ReAct-style agent, pass --agent react to pipeline.py (other settings stay the same).

Docker

./build-docker.sh

Run a filesystem task (no external accounts required):

python -m pipeline \
  --mcp filesystem \
  --k 1 \ # run once to quick start
  --models gpt-5  \ # or any model you configured
  --tasks file_property/size_classification
# Add --task-suite easy to run the lightweight dataset (where available)

Results are saved to ./results/{exp_name}/{model}__{mcp}/run-*/... for the standard suite and ./results/{exp_name}/{model}__{mcp}-easy/run-*/... when you run --task-suite easy (e.g., ./results/test-run/gpt-5__filesystem/run-1/... or ./results/test-run/gpt-5__github-easy/run-1/...).

---

Run your evaluations

Task suites (standard vs easy)

• Each MCP service now stores tasks under tasks/<mcp>/<task_suite>/<category>/<task>/.
• standard (default) covers the full benchmark (127 tasks today).
• easy hosts 10 lightweight tasks per MCP, ideal for smoke tests and CI (GitHub’s are already available under tasks/github/easy).
• Switch suites with --task-suite easy (defaults to --task-suite standard).

Single run (k=1)

# Run ALL tasks for a service
python -m pipeline --exp-name exp --mcp notion --tasks all --models MODEL --k 1

# Run a task group
python -m pipeline --exp-name exp --mcp notion --tasks online_resume --models MODEL --k 1

# Run a specific task
python -m pipeline --exp-name exp --mcp notion --tasks online_resume/daily_itinerary_overview --models MODEL --k 1

# Evaluate multiple models
python -m pipeline --exp-name exp --mcp notion --tasks all --models MODEL1,MODEL2,MODEL3 --k 1

Multiple runs (k>1) for pass@k

# Run k=4 to compute stability metrics (requires --exp-name to aggregate final results)
python -m pipeline --exp-name exp --mcp notion --tasks all --models MODEL

# Aggregate results (pass@1 / pass@k / pass^k / avg@k)
python -m src.aggregators.aggregate_results --exp-name exp

Run with Docker

# Run all tasks for a service
./run-task.sh --mcp notion --models MODEL --exp-name exp --tasks all

# Cross-service benchmark
./run-benchmark.sh --models MODEL --exp-name exp --docker

Please visit docs/introduction.md for choices of MODEL.

Tip: MCPMark supports auto-resume. When re-running, only unfinished tasks will execute. Failures matching our retryable patterns (see RETRYABLE_PATTERNS) are retried automatically. Models may emit different error strings—if you encounter a new resumable error, please open a PR or issue.

Tip: MCPMark supports auto-compaction; pass --compaction-token N to enable automatic context summarization when prompt tokens reach N (use 999999999 to disable).

---

Service setup and authentication

You can also follow Quickstart for the shortest end-to-end path.

Important Notice: GitHub Repository Privacy

Please ensure your evaluation repositories are set to PRIVATE.

GitHub state templates are now automatically downloaded from our CDN during evaluation — no manual download is required. However, because these templates contain issues and pull requests from real open-source repositories, the recreation process includes @username mentions of the original authors.

We have received feedback from original GitHub authors who were inadvertently notified when evaluation repositories were created as public. To be a responsible member of the open-source community, we urge all users to:

1. Always keep evaluation repositories private during the evaluation process.
2. In the latest version, we have added random suffixes to all @username mentions (e.g., @user becomes @user_x7k2) and implemented a safety check that prevents importing templates to public repositories.
3. If you are using an older version of MCPMark, please either:
   • Pull the latest code immediately, or
   • Manually ensure all GitHub evaluation repositories are set to private.

Thank you for helping us maintain a respectful relationship with the open-source community.

---

Results and metrics

• Results are organized under ./results/{exp_name}/{model}__{mcp}/run-*/ (JSON + CSV per task).
• Generate a summary with:

# Basic usage
python -m src.aggregators.aggregate_results --exp-name exp

# For k-run experiments with single-run models
python -m src.aggregators.aggregate_results --exp-name exp --k 4 --single-run-models claude-opus-4-1

• Only models with complete results across all tasks and runs are included in the final summary.
• Includes multi-run metrics (pass@k, pass^k) for stability comparisons when k > 1.

---

Model and Tasks

• Model support: MCPMark calls models via LiteLLM — see the LiteLLM docs: LiteLLM Doc. For Anthropic (Claude) extended thinking mode (enabled via --reasoning-effort), we use Anthropic’s native API.
• See docs/introduction.md for details and configuration of supported models in MCPMark.
• To add a new model, edit src/model_config.py. Before adding, check LiteLLM supported models/providers. See LiteLLM Doc.
• Task design principles in docs/datasets/task.md. Each task ships with an automated verify.py for objective, reproducible evaluation, see docs/task.md for details.

---

Contributing

Contributions are welcome:

1. Add a new task under tasks/<mcp>/<task_suite>/<category_id>/<task_id>/ with meta.json, description.md and verify.py.
2. Ensure local checks pass and open a PR.
3. See docs/contributing/make-contribution.md.

---

Citation

If you find our works useful for your research, please consider citing:

@misc{wu2025mcpmark,
      title={MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use}, 
      author={Zijian Wu and Xiangyan Liu and Xinyuan Zhang and Lingjun Chen and Fanqing Meng and Lingxiao Du and Yiran Zhao and Fanshi Zhang and Yaoqi Ye and Jiawei Wang and Zirui Wang and Jinjie Ni and Yufan Yang and Arvin Xu and Michael Qizhe Shieh},
      year={2025},
      eprint={2509.24002},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.24002}, 
}

License

This project is licensed under the Apache License 2.0 — see LICENSE.