SMITH-Exp

SMITH-Exp

SMITH-Exp is an instruction-tuned Mixture-of-Experts checkpoint based on GigaChat 3.1 Lightning (10B total parameters, 1.8B active, BF16). It is optimized for multi-hop retrieval through GigaChat 3 tool calls: the model iteratively invokes search_index, then returns an answer with supporting_corpus_ids.

For reproducible results, use the checkpoint with the SMITH retrieval harness or an equivalent implementation of the same tool-calling loop. The evaluation results below assume this execution protocol; behavior and quality are not guaranteed in general-purpose chat frameworks or integrations that use different prompts, tools, parsing, or stopping logic.

The included gigachat3_guided_decoding.py plugin integrates with vLLM's GigaChat 3 tool parser and constrains tool calls to the expected structure. This reduces malformed calls and allows the harness to parse search_index and final answer actions reliably. Use the plugin with the supplied chat template, as shown below.

Evaluation

MuSiQue answerable validation (bdsaglam/musique), 100 queries mixing the three four-hop topologies: 50 4hop1, 25 4hop2, and 25 4hop3. Ranking is against gold supporting paragraphs at k=10 on a text-deduplicated train+validation FAISS index. Agent runs use temperature 0, retriever_k=100, agent_max_k=10, and at most six searches.

Citation scores the supporting_corpus_ids in the final answer. Search scores the union of documents retrieved over the trajectory. Searches is the mean number of search_index calls per question.

The embedder-only row is a comparison baseline (single top-10 search of the original question, no agent). Agents still call search_index as a tool; that tool uses Qwen3-Embedding-4B (2,560-d).

<table> <thead> <tr> <th align="left">Model</th> <th align="right">nDCG@10 (citation)</th> <th align="right">nDCG@10 (search)</th> <th align="right">Recall@10 (citation)</th> <th align="right">Recall@10 (search)</th> <th align="right">Searches</th> </tr> </thead> <tbody> <tr><td colspan="6"><strong>Agents</strong></td></tr> <tr> <td><strong>SMITH-Exp</strong></td> <td align="right">0.664</td> <td align="right">0.766</td> <td align="right">0.629</td> <td align="right">0.680</td> <td align="right">5.73</td> </tr> <tr> <td>GLM-5.3</td> <td align="right">0.611</td> <td align="right">0.709</td> <td align="right">0.515</td> <td align="right">0.610</td> <td align="right">4.64</td> </tr> <tr> <td>DeepSeek-V4-Flash-0731</td> <td align="right">0.591</td> <td align="right">0.726</td> <td align="right">0.523</td> <td align="right">0.638</td> <td align="right">5.84</td> </tr> <tr> <td>Qwen3.6-35B-A3B</td> <td align="right">0.546</td> <td align="right">0.712</td> <td align="right">0.445</td> <td align="right">0.620</td> <td align="right">5.57</td> </tr> <tr><td colspan="6"><strong>Embedders</strong></td></tr> <tr> <td>Qwen3-Embedding-4B</td> <td align="right">0.333</td> <td align="right">0.333</td> <td align="right">0.300</td> <td align="right">0.300</td> <td align="right">—</td> </tr> </tbody> </table>

Usage

Serving dependencies

uv venv --python python3.11 --seed .venv
uv pip install -r requirements.txt \
  --python .venv/bin/python \
  --torch-backend=cu130 \
  --index-strategy unsafe-best-match

Package versions and license information are listed in requirements.txt. The model fits on a single 80 GB GPU.

transformers

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

model_name = "ai-forever/SMITH-Exp"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.generation_config = GenerationConfig.from_pretrained(model_name)
messages = [
    {"role": "user", "content": "Which corpus passages support the answer?"}
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=512)
prompt_len = inputs["input_ids"].shape[1]
print(tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True))

vLLM

vllm serve ai-forever/SMITH-Exp \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser gigachat3 \
  --tool-parser-plugin ./gigachat3_guided_decoding.py \
  --chat-template ./chat_template.jinja \
  --tensor-parallel-size 1 \
  --dtype bfloat16

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai-forever/SMITH-Exp",
    "temperature": 0,
    "tool_choice": "required",
    "messages": [
      {"role": "user", "content": "Which corpus passages support the answer?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "search_index",
          "description": "Search the local semantic index and return relevant text snippets.",
          "parameters": {
            "type": "object",
            "properties": {
              "query": {"type": "string", "description": "The search query string."},
              "k": {"type": "integer", "description": "Optional number of top results to return."}
            },
            "required": ["query"]
          }
        }
      }
    ]
  }'

Authors

• Anton Chernov (GitHub)
• Vladislav Leshchuk (GitHub)
• Artemiy (GitHub)
• Artem Snegirev (GitHub)