crab-1

Crab-1 — a 1.7B OSINT agent you can actually test

Crab-1 is a Qwen3-1.7B fine-tuned (QLoRA SFT) to build a factual profile of a French company from its name alone — official website, administrative department, sector — using three tools: the official French company registry, web search, and page extraction. It runs locally on a consumer GPU, costs nothing per profile, and finishes in ~4 tool-calling turns (~9 s/profile on an RTX 3060 Ti).

Everything needed to verify the published claims is public: the evaluation harness, the 30-company ground-truth set, and the scoring function live in the GitHub repository. The ground-truth set is also published as a dataset: gaidar12345/crab-1-benchmark. The training-data factory (teacher generation, trajectory filtering, dataset build) is deliberately not published.

Files

Published numbers — with dates, because they matter

Pass rate = reward ≥ 0.7 (TLD-strict website match + department-level location, dynamic denominator — see harness/reward.py in the GitHub repo).

⚠️ This benchmark is perishable. The tools hit the live web. The same weights scored 76.7 % in July and 66.7 % in late August — nothing changed but the search results. The untrained baseline moved even more (56.7 % → 70.0 %). Two consequences:

1. Only compare models measured the same day.
2. Your numbers will differ from ours. Expect the ballpark, not the digit.

What stays stable across dates: Crab-1 always submits a profile within the turn budget (100 % submit rate vs 60–80 % for the frontier models, which tend to keep verifying past the 8-turn cap), and it's 2–4× faster end-to-end. Fine-tuning didn't buy intelligence — it bought protocol discipline: stop on time, fill the exact schema. For a fleet of cheap local crawlers, that's the property that matters.

Out-of-distribution check (2026-09-01, 20 known companies)

Because the 30-company benchmark is all startup-style companies, we also ran 20 well-known French companies the model never saw in any form — CAC40 groups, regional SMEs, tricky brand names (Michelin, Airbus, Back Market, Fermob, Saint James…). Set + raw results: data/ood_*.json in the GitHub repo; run your own list with --eval-file.

• Website finding generalizes: 0.78 accuracy, most partial misses are defensible TLD variants (michelin.fr vs .com).
• Location does not — for either model (0.26 for both, vs 0.87–0.97 on the startup benchmark). The cause is the tool, not the fine-tune: registry lookup by name returns the wrong entity for famous brands.
• Pass rate is a statistical tie — out of distribution the fine-tune behaves like the base with the same protocol discipline.
• Known single failure worth naming: "Armor Lux" → the model invented a hyphen (armor-lux.com does not exist). The base found the right domain. That's the class of error to expect from a 1.7B.

Usage

Ollama (recommended)

# 1. Get the weights, then from the GitHub repo:
ollama create crab1-v7 -f Modelfile

# 2. One profile, interactively:
python quickstart.py "Doctolib"

# 3. The full 30-company eval (~5 min, live web):
python eval/run_eval.py --model crab1-v7 --out results/crab1_v7.json

Transformers (LoRA adapter)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B")
model = PeftModel.from_pretrained(base, "gaidar12345/crab-1/lora-adapter")

Honest limitations

• Trained and evaluated on French companies only; the registry tool is France-specific.
• The eval set is 30 companies — big enough to rank models, too small for decimal-point bragging.
• Name collisions in the registry (brand name ≠ legal name) are the main remaining failure mode.
• The model is a specialist. Ask it anything outside "profile this French company with these tools" and the base 1.7B is what you get.

License

Weights: Apache-2.0 (derivative of Qwen3-1.7B). Code in the GitHub repo: MIT.

Write-up with the full story: https://yegorgaidar.org/blog/crab1-slm-osint/