fugu-lite

BenchGen Fugu-Lite

Fugu-Lite is a small, runnable model router for one local GPU. A frozen local language model reads a task, a custom linear head selects one worker, and the selected worker is called through OpenRouter (or through a local OpenAI-compatible server).

This is Fugu-inspired research code, not Sakana AI's proprietary Fugu implementation. The useful overlap is the learned worker selection, offline trajectory/reward collection, and an optional separable CMA-ES stage. Version 0.1 performs one learned routing decision; it does not yet learn recursive, multi-turn delegation or answer synthesis.

Project documentation

• How the system works
• V1 experiment and progress record
• Use the private Hugging Face repository
• Documentation index

The repository includes the V1 task/reward snapshot and trained routing-head artifacts so the experiment can be audited and resumed. The real API key is never stored in the project.

What trains where

The default backbone is Qwen/Qwen3-0.6B. With three workers, its trainable head is only a few thousand parameters. The backbone stays frozen, so the project is practical on modest GPUs. The exact memory depends on sequence length and software, but the default normally fits comfortably above roughly 4 GiB of VRAM.

Architecture

flowchart TD
    A[BenchGen tasks] --> B[Call every worker once]
    B --> C[Quality, cost, latency rewards]
    C --> D[Reward matrix JSONL]
    D --> E[SFT warm-start]
    E --> F[Contextual-bandit RL]
    F --> G[Optional Sep-CMA-ES]
    G --> H[Local router checkpoint]
    H --> I[Single API endpoint]
    I --> J[Selected OpenRouter or local worker]

1. Install on the remote GPU server

Python 3.10–3.13 and a working NVIDIA/PyTorch setup are expected.

Install the PyTorch CUDA build recommended for your server's driver from https://pytorch.org/get-started/locally/ first. If python -c "import torch; print(torch.cuda.is_available())" already prints True, keep that installation.

unzip BenchGen_Fugu_Lite.zip
cd fugu-lite
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e '.[dev]'
cp .env.example .env
chmod 600 .env

Edit .env on the server and set OPENROUTER_API_KEY. Never paste the real key into source code, YAML, a Git commit, or chat. Check the machine:

fugu-lite doctor

If PyTorch cannot see CUDA, fix the server's NVIDIA driver/PyTorch installation before training. The router can run on CPU, but it will be much slower.

2. Prove the whole data path without spending credits

The mock pool contains a math, code, and general expert. It deterministically creates a reward matrix from the sample tasks.

fugu-lite generate \
  --workers configs/workers.mock.yaml \
  --tasks data/tasks.sample.jsonl \
  --output data/rewards.mock.jsonl

Run the tests:

pytest -q

3. Choose real OpenRouter workers

OpenRouter model availability changes, so query the live catalog rather than relying on an old list:

fugu-lite list-models --contains qwen --limit 30
fugu-lite list-models --contains free --limit 30

Copy and edit the worker file:

cp configs/workers.openrouter.example.yaml configs/workers.yaml

Choose meaningfully different workers—for example, a low-cost fast model, a strong reasoning model, and a coding specialist. If all workers produce the same reward on every task, the router has no learning signal. Three or four workers are a good first experiment.

To include a model hosted on the same server, add an openai_compatible worker. Any server that exposes /v1/chat/completions can be used; an example vLLM entry is already commented in the YAML.

4. Prepare real BenchGen tasks

Input is JSON Lines, one object per task:

{"task_id":"math-101","prompt":"Solve ...","domain":"math","reference_answer":"42","grader":{"type":"numeric","tolerance":0.0},"split":"train","tags":["algebra"]}

Supported local graders are exact, contains, numeric, and regex. For an open-ended task, use llm_judge, add a clear grader.rubric, and configure reward.judge_model in the worker YAML. LLM judging adds one judge call for each worker response and should be audited on a human-labeled sample.

Do not place answers, benchmark labels, or worker scores in prompt; only the grader sees the reference answer. Keep benchmark test sets out of training. A useful first real run is:

• 200–1,000 training tasks covering your production distribution;
• at least 50 validation and 100 held-out test tasks;
• balanced domains, including tasks on which each worker can actually win;
• deterministic executable graders whenever possible.

If BenchGen already produces worker rewards, skip API generation and write the matrix shape shown in data/rewards.schema.example.jsonl. worker_ids order must remain identical in every row, and rewards[i] must belong to worker_ids[i].

5. Generate the real reward matrix

First use a small limit to verify prompts, grading, and billing:

fugu-lite generate \
  --workers configs/workers.yaml \
  --tasks data/tasks.jsonl \
  --output data/rewards.real.jsonl \
  --limit 10

Inspect the JSONL. When it looks correct, run the complete job (preferably to a new output path). Records are appended after each task, so an interrupted job can continue safely with --resume. The number of primary calls is number_of_tasks × number_of_workers; llm_judge doubles that count. OpenRouter's non-streaming usage object is recorded when it provides token and cost fields.

fugu-lite generate \
  --workers configs/workers.yaml \
  --tasks data/tasks.jsonl \
  --output data/rewards.real.jsonl \
  --resume

The default utility is:

[ U = w_q Q - w_c \min(C/C_0,5) - w_l \min(L/L_0,5) ]

where quality (Q) is 0–1, cost (C) is the reported request cost, and latency (L) is seconds. Adjust the weights and scales in the worker YAML to match your product. If quality matters much more than price, keep the cost and latency weights small.

6. Train the routing head

Start with SFT. It converts each reward vector into a soft target distribution and trains the custom head with cross-entropy:

fugu-lite train-sft \
  --config configs/train_sft.yaml \
  --data data/rewards.real.jsonl \
  --output artifacts/router-sft

Then run real offline contextual-bandit RL. The default REINFORCE estimator samples worker actions and uses the full-information expected reward as a baseline:

fugu-lite train-rl \
  --config configs/train_rl.yaml \
  --data data/rewards.real.jsonl \
  --checkpoint artifacts/router-sft \
  --output artifacts/router-rl

For smaller or noisy data, estimator: expected_reward in train_rl.yaml is lower variance. For a closer analogue to the reported Sakana optimization style, optionally run separable CMA-ES. It caches backbone features once and evolves only the last linear layer:

fugu-lite train-es \
  --config configs/train_es.yaml \
  --data data/rewards.real.jsonl \
  --checkpoint artifacts/router-rl \
  --output artifacts/router-es

Use SFT alone as the first baseline. Add RL only when workers have different rewards and the validation utility improves. Add CMA-ES when you want to optimize a discrete or otherwise non-differentiable end metric; keep it only if held-out performance improves.

7. Evaluate correctly

fugu-lite evaluate \
  --checkpoint artifacts/router-rl \
  --data data/rewards.real.jsonl \
  --split test \
  --output artifacts/router-rl/test_metrics.json

The report includes:

Do not deploy merely because route accuracy looks high. The router should beat the best fixed worker on held-out utility and on your actual cost/latency guardrails.

8. Use the trained router

Route without calling any worker:

fugu-lite route \
  --checkpoint artifacts/router-rl \
  --domain code \
  --prompt "Find the bug in this Python function ..."

Route and call the selected worker:

fugu-lite ask \
  --checkpoint artifacts/router-rl \
  --workers configs/workers.yaml \
  --domain code \
  --prompt "Find the bug in this Python function ..."

Serve a single OpenAI-compatible endpoint:

fugu-lite serve \
  --checkpoint artifacts/router-rl \
  --workers configs/workers.yaml \
  --host 0.0.0.0 \
  --port 8080

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"fugu-lite","domain":"code","messages":[{"role":"user","content":"Explain this stack trace..."}]}'

The response includes a fugu_lite field with route probabilities, the actual served model, latency, cost when available, and fallback errors. In production, put authentication and TLS in front of this service; do not expose it directly to the public internet.

What to improve next

Version 0.1 is deliberately narrow. The next meaningful step toward a fuller Fugu-like system is to define a small action grammar such as CALL(worker, subtask), VERIFY(worker), SYNTHESIZE, and STOP; collect complete multi-turn trajectories in a sandbox; and optimize the final verified task reward plus cost/latency penalties. Do that only after this one-step router reliably beats fixed-worker baselines—otherwise multi-agent complexity hides basic data and reward problems.

References

• Sakana AI Fugu release: https://sakana.ai/fugu-release/
• OpenRouter quickstart: https://openrouter.ai/docs/quickstart
• OpenRouter usage accounting: https://openrouter.ai/docs/cookbook/administration/usage-accounting
• Qwen3-0.6B model card: https://huggingface.co/Qwen/Qwen3-0.6B
• cmaes Sep-CMA implementation: https://github.com/CyberAgentAILab/cmaes