Tool-Genesis-Qwen3-8B-SFT

Tool-Genesis-Qwen3-8B-SFT

A fine-tuned Qwen3-8B model for autonomous MCP (Model Context Protocol) tool server generation. Given a natural language scenario description, the model generates a complete, runnable MCP server with tool schemas and implementation code.

Model Details

Training

The model was fine-tuned on curated MCP server generation examples from the Tool-Genesis benchmark. Each training sample consists of:

• Input: A natural language scenario description specifying what the MCP server should do
• Output: A complete Python MCP server implementation using the FastMCP framework

Training Configuration

• Epochs: 3
• Total steps: 117 (~39 steps/epoch)
• Final training loss: 0.522
• Training runtime: ~4.6 hours

Loss Curve

Benchmark Results

Evaluated on the Tool-Genesis Benchmark (86 MCP servers, 4-level evaluation).

Direct Generation (single-call, no agent loop)

SFT gains over base Qwen3-8B (Direct):

• L1 Compliance: +14.0% (0.686 → 0.826)
• L1 Launch: +3.5% (0.012 → 0.047)
• L2 Schema F1: +3.5% (0.011 → 0.046)
• L2 UT Soft: +1.6% (0.001 → 0.017)

With Coder-Agent (multi-turn with sandbox)

Note: The coder-agent strategy dramatically improves all models by providing an iterative sandbox-based coding loop. The SFT model has not yet been evaluated with the coder-agent strategy.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "tool-genesis/Tool-Genesis-Qwen3-8B-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

prompt = """You are a developer building MCP tool servers in Python.
Build a complete MCP server for the following scenario:

A weather information service that provides current weather data, 
forecasts, and weather alerts for any location worldwide.

Output only the Python source code using the FastMCP framework."""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=4096, temperature=0.2)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation Protocol

The Tool-Genesis benchmark evaluates generated MCP servers across four levels:

Links

• Benchmark Dataset: tool-genesis/Tool-Genesis-Benchmark
• Code: github.com/Tool-Genesis/Tool-Genesis

Citation

@misc{tool_genesis_2025,
  title={Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent},
  author={Xia, Bowei and Hu, Mengkang and Wang, Shijian and Jin, Jiarui and Jiao, Wenxiang and Lu, Yuan and Li, Kexin and Luo, Ping},
  year={2025},
  note={Project page: https://tool-genesis.github.io}
}

License

Apache 2.0