Fathom-Search-4B-GGUF

<span style="color: #7FFF7F;">Fathom-Search-4B GGUF Models</span>

<span style="color: #7F7FFF;">Model Generation Details</span>

This model was generated using llama.cpp at commit 03792ad93.

---

<span style="color: #7FFF7F;">Quantization Beyond the IMatrix</span>

I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.

In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the --tensor-type option in llama.cpp to manually "bump" important layers to higher precision. You can see the implementation here:
👉 Layer bumping with llama.cpp

While this does increase model file size, it significantly improves precision for a given quantization level.

I'd love your feedback—have you tried this? How does it perform for you?

---

<a href="https://readyforquantum.com/huggingface_gguf_selection_guide.html" style="color: #7FFF7F;"> Click here to get info on choosing the right GGUF model format </a>

---

<!--Begin Original Model Card-->

Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval And Synthesis For SLMs

This model was presented in the paper Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs.

<div align="center">

</div> <div align="center">

</div> <div align="center">

</div> <div align="center">

</div>

---

<!-- <p align="center"> <img src="./assets/Final-WebWalker.png" style="width: 100%;" id="benchmark-img"> </p> <p align="center"> <img src="./assets/Final-SimpleQA.png" style="width: 100%;" id="benchmark-img"> </p> --> <p align="center"> <img src="https://raw.githubusercontent.com/FractalAIResearchLabs/Fathom-DeepResearch/refs/heads/main/assets/Final-DeepResearch.png" style="width: 50%;" id="benchmark-img"> </p> <p align="center"> <img src="https://raw.githubusercontent.com/FractalAIResearchLabs/Fathom-DeepResearch/refs/heads/main/assets/deep-search.png" style="width: 100%;" id="benchmark-img"> </p>

---

Introduction

We introduce Fathom-DeepResearch, an agentic DeepResearch system that sets state-of-the-art performance in the open-weights category on search-intensive benchmarks (SimpleQA, FRAMES, WebWalkerQA, Seal0) and outperforms closed-source DeepResearch agents like (Claude, Grok, Perplexity, GPT-4o) on open-ended synthesis benchmark DeepResearch-Bench. Built on two specialized models, Fathom-Search-4B, optimized for long-horizon, evidence-seeking through live web search, and Fathom-Synthesizer-4B, designed for open-ended synthesis and report generation. Refer to our technical report for our training recipe and more exciting results!

<p align="center"> <img src="https://raw.githubusercontent.com/FractalAIResearchLabs/Fathom-DeepResearch/refs/heads/main/assets/main.png" style="width: 100%;" id="benchmark-img"> </p>

🤗 Key Release Assets

• 🤗 Fathom-Search-4B & 🤗 Fathom-Synthesizer-4B
  We opensource two 4B-parameter models trained to browse, extract, verify and reason over live web content, and synthesize the findings into rich citation-dense reports!

• 🌐 Fathom-WebAgents
  We open-source our search tool server, built on top of Jina-AI, Crawl4AI, Trafilatura, and Serper.dev, which acts as a powerful search backend, optimized to handle large volumes of asynchronous requests and includes specialized handlers for diverse sources such as YouTube, PDFs, Reddit, Github, and more.

• 🤗 DuetQA
  We release a ~5K-sample dataset created through our novel multi-agent self-play pipeline, containing verifiable question–answer pairs that are impossible to answer without live web search , for DeepSearch model training.

• 🤗 DeepResearch-SFT
  We are releasing a synthetic SFT corpus used to train Fathom-Syntheizer-4B, to enable the conversion of downstream search/investigation traces of DeepSearch enabled models into DeepResearch reports via our novel plan-then-write protocol.

• 📚 Technical Report
  We release technical report outlining key training innovations, dataset generation pipeline, complete training startegy and evaluation!

✨ Key Innovations

• 🧭 Multi-Agent Self-Play
  We present a self-supervised dataset construction framework designed to yield verifiable, live web-search enforcing, multi-hop QA pairs. This pipeline serves as the basis for generating DUETQA, a dataset tailored for training agentic deepsearch models.

• ⚡ RAPO: Reward-Aware Policy Optimization
  We introduce RAPO, a zero-overhead extension of GRPO that stabilizes multi-turn Reinforcement Learning with Verifiable Rewards through curriculum pruning, reward-aware advantage scaling, and per-prompt replay buffers that stabilize long-horizon tool use.

• 🎯 Steerable Step-Level Reward
  We design our novel Steerable Step-Level Reward that alleviates the reward-hacking challenge faced by RLVR training in the multi-turn, tool-interaction. Our reward function enables us to steer (i) how much the agent uses tools and (ii) how it allocates cognition to exploration and verification.

• 📝 DeepResearch Report Synthesis Protocol
  We propose a novel plan-then-write protocol for synthesizing DeepSearch traces into citation-dense reports, where planning involves decomposition of questions into ordered sub-questions, evidence-to-section mapping aligning URLs to the correct sub-question, and insight planning to guide synthesis into structured reports.

---

📊 Key Results

<p align="center"> <img src="https://raw.githubusercontent.com/FractalAIResearchLabs/Fathom-DeepResearch/refs/heads/main/assets/benchmarks.png" style="width: 100%;" id="benchmark-img"> </p> <p align="center"> <img src="https://raw.githubusercontent.com/FractalAIResearchLabs/Fathom-DeepResearch/refs/heads/main/assets/deepresearchbench.png" style="width: 100%;" id="deepresearchbench-img"> </p>

🚀 Inference

This section shows how to host Fathom-DeepResearch locally and run inference!.

1) Environment setup

uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126
uv pip install --no-build-isolation -r requirements.txt
playwright install-deps
playwright install

2) Start the Model Servers (SGLang)

Change --model-path to your model identifier (eg. FractalAIResearch/Fathom-Search-4B) or local path.

Fathom-Search-4B

Launch on default port below 8902, and GPU 0

export CUDA_VISIBLE_DEVICES=0
python -m sglang.launch_server \
          --served-model-name  Fathom-Search-4B \
          --model-path FractalAIResearch/Fathom-Search-4B \
          --enable-metrics \
          --dtype bfloat16 \
          --host 0.0.0.0 \
          --port 8902 \
          --trust-remote-code \
          --disable-radix-cache \
          --disable-cuda-graph \
          --context-length 40960 \
          #--tp 2 #optional for multi-gpu inference

Fathom-Synthesizer-4B

Launch on default port 8903 and GPU 1, we use YARN scaling with RoPE factor 2.0 for the synthesizer model

export CUDA_VISIBLE_DEVICES=1
python3 -m sglang.launch_server \
         --served-model-name Fathom-Synthesizer-4B \
         --model-path FractalAIResearch/Fathom-Synthesizer-4B \
         --enable-metrics \
         --dtype bfloat16 \
         --host 0.0.0.0 \
         --port 8903 \
         --trust-remote-code \
         --disable-radix-cache \
         --disable-cuda-graph \
         --disable-cuda-graph \
         --context-length 131092 \
         --json-model-override-args '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":40960}' \
        #--tp 2 #optional for multi-gpu inference

(Optional) Query-LLM for search backend

Launch on default port 8905 and GPU 3

export CUDA_VISIBLE_DEVICES=3
python3 -m sglang.launch_server \
         --served-model-name Query-LLM \
         --model-path Qwen3/Qwen3-32B #Any LLM of choice to act as the query LLM for search tool server \
         --enable-metrics \
         --dtype bfloat16 \
         --host 0.0.0.0 \
         --port 8905 \
         --trust-remote-code \
         --disable-radix-cache \
         --disable-cuda-graph \
         --disable-cuda-graph \
         --context-length 131092 \
        #--tp 2 #optional for multi-gpu inference

3) Fire up the Search Tools Server (Serper + Jina + Query LLM)

<p align="center"> <img src="https://raw.githubusercontent.com/FractalAIResearchLabs/Fathom-DeepResearch/refs/heads/main/assets/search.png" style="width: 100%;" id="benchmark-img"> </p>

Set the following in scripts/.env:

• SERPER_API_KEY (get from serper.dev; ~2,500 free queries without any card) (necessary for live web-search)
• JINA_API_KEY (optional) — used in the web-page extraction pipeline (recommended for replicaiion)
• OPENAI_API_KEY (optional) — for goal conditioned querying of web-pages using GPT-4.1-mini (recommended for replication)
• SERPER_CACHE_DIR — path to caching the search results from serper.dev to save cost and retrival time
• JINA_CACHE_DIR — path to caching the search results from jina.ai to save cost and retrival time

(Recommended) Launch search tool server on port 8904 with 256 workers with gpt-4.1-mini as the Query-LLM (OpenAI API key needed):

serving/host_server.sh 8904 256 "openai:gpt-4.1-mini"

(Or) Launch search tool server on port 8904 with a locally hosted LLM of choice via SGLang at port 8905 (See step 2) as the Query-LLM:

serving/host_server.sh 8904 256 "http://0.0.0.0:8905"

Note: Our search backend also supports web_search and web_visit functions for searching the web and returning entire web-page w/o the need for a Query-LLM. Used by model like Jan-Naao and II-Search-4B in their implementation.

4) Run DeepResearch! (Single question inference)

Run Fathom‑DeeoSearch via inference.py:

 python inference.py \
    --question "Find out the names of all the companies that have been selected under the IndiaAI mission? Write a detailed report on what their deliverables are? What are the ways in which this mission will impact india in the coming decade?" \
    --executors http://0.0.0.0:8904 \
    --model-url http://0.0.0.0:8902 \
    --tokenizer FractalAIResearch/Fathom-Search-4B \
    --summary-llm http://0.0.0.0:8903 \
    --deepresearch #DeepResearch mode!, disable to get Fathom-Search-4B response Faster & shorter

Tips:

• Use multiple executors for load‑balancing: --executors http://0.0.0.0:8904, http://0.0.0.0:8905.

---

🧮 Evaluation (Multi GPU)

This section covers batched evaluation using the provided scripts in scripts/. Use placeholders model_path and dataset_name — the evaluator will read eval_benchmarks/<dataset_name>.jsonl with columns ['id','question','answer'].

Common flags

Evaluate Fathom‑Search

GPT-4.1-mini query-LLM on CPU, main model on GPUs 0,1 (TP=2)

scripts/eval_fathom_search.sh \
  --model-path FractalAIResearch/Fathom-Search-4B \
  --model-port 1255 \
  --executors 1211,1212 \
  --dataset dataset_name \
  --main-gpus 0,1 \
  --query-llm gpt-4.1-mini

Local Qwen3-32B as Query-LLM on GPUs 2,3 (TP=2); main model on GPUs 0,1 (TP=2)

scripts/eval_fathom_search.sh \
  --model-path FractalAIResearch/Fathom-Search-4B \
  --model-port 1255 \
  --executors 1211,1212 \
  --dataset dataset_name \
  --main-gpus 0,1 \
  --query-llm Qwen3/Qwen3-32B \
  --query-port 1260 \
  --query-gpus 2,3

Evaluate other baselines used in the paper

# II‑Search‑4B
scripts/eval_ii_search.sh \
  --model-path Intelligent-Internet/II-Search-4B \
  --model-port 1255 \
  --executors 1211,1212 \
  --dataset dataset_name

# Jan‑Nano‑32K
scripts/eval_jan_nano_32K.sh \
  --model-path Menlo/Jan-nano \
  --model-port 1255 \
  --executors 1211,1212 \
  --dataset dataset_name

# Jan‑Nano‑128K
scripts/eval_jan_nano_128K.sh \
  --model-path Menlo/Jan-nano-128k \
  --model-port 1255 \
  --executors 1211,1212 \
  --dataset dataset_name

# R1‑Searcher(7B)
scripts/eval_r1_searcher.sh \
  --model-path XXsongLALA/Qwen-2.5-7B-base-RAG-RL \
  --model-port 1255 \
  --dataset dataset_name

# ZeroSearch
scripts/eval_zerosearch.sh \
  --model-path Alibaba-NLP/ZeroSearch_wiki_V2_Qwen2.5_7B_Instruct \
  --model-port 1255 \
  --dataset dataset_name

# search‑o1 (with Qwen3‑4B)
scripts/eval_search_o1.sh \
  --model-path Qwen3/Qwen3-4B \
  --model-port 1255 \
  --dataset dataset_name

#cleanup ports after eval 
 scripts/cleanup.sh 1255 1211 1212

Run

---

📜 License

This repository and all the release assets are available under the MIT License, underscoring our dedication to open and inclusive AI innovation. By freely sharing our work, we aim to democratize AI technology, empowering researchers, developers, and enthusiasts everywhere to use, adapt, and expand upon it without limitation. This open and permissive approach promotes global collaboration, accelerates innovation, and enriches the AI community as a whole.

Acknowledgments

We would like to acknowledge the following works for enabling our project:

• Qwen3-4B
• ReCall

---

📖 Citation

@misc{fathomdeepsearch2025,
  title={Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval And Synthesis For SLMs},
  author={Shreyas Singh and Pradeep Moturi and Kunal Singh},
  howpublished={\url{https://huggingface.co/FractalAIResearch/Fathom-Search-4B}},
  note={Hugging Face},
  year={2025}
}

---

<!--End Original Model Card-->

---

<span id="testllm" style="color: #7F7FFF;">🚀 If you find these models useful</span>

Help me test my AI-Powered Quantum Network Monitor Assistant with quantum-ready security checks:

👉 Quantum Network Monitor

The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : Source Code Quantum Network Monitor. You will also find the code I use to quantize the models if you want to do it yourself GGUFModelBuilder

💬 How to test:
Choose an AI assistant type:

• TurboLLM (GPT-4.1-mini)
• HugLLM (Hugginface Open-source models)
• TestLLM (Experimental CPU-only)

What I’m Testing

I’m pushing the limits of small open-source models for AI network monitoring, specifically:

• Function calling against live network services
• How small can a model go while still handling:
  • Automated Nmap security scans
  • Quantum-readiness checks
  • Network Monitoring tasks

🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):

• ✅ Zero-configuration setup
• ⏳ 30s load time (slow inference but no API costs) . No token limited as the cost is low.
• 🔧 Help wanted! If you’re into edge-device AI, let’s collaborate!

Other Assistants

🟢 TurboLLM – Uses gpt-4.1-mini :

• **It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
• Create custom cmd processors to run .net code on Quantum Network Monitor Agents
• Real-time network diagnostics and monitoring
• Security Audits
• Penetration testing (Nmap/Metasploit)

🔵 HugLLM – Latest Open-source models:

• 🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.

💡 Example commands you could test:

1. "Give me info on my websites SSL certificate"
2. "Check if my server is using quantum safe encyption for communication"
3. "Run a comprehensive security audit on my server"
4. '"Create a cmd processor to .. (what ever you want)" Note you need to install a Quantum Network Monitor Agent to run the .net code on. This is a very flexible and powerful feature. Use with caution!

Final Word

I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is open source. Feel free to use whatever you find helpful.

If you appreciate the work, please consider buying me a coffee ☕. Your support helps cover service costs and allows me to raise token limits for everyone.

I'm also open to job opportunities or sponsorship.

Thank you! 😊