Skill Arena

Skill Arena

Test play for skills and deck configurations. Not "which is best" — "which is best for what."

Decision Tree (READ FIRST)

User says: "test/​compare/​arena/​benchmark/​A vs B"
    │
    ├── Cross-PLAYER? (kimi vs codex vs claude)
    │   OR user explicitly says useAgent/​specific player
    │   OR platform doesn't support Agent tool subagents
    │     → CLI runner REQUIRED (useAgent → Bun.spawn)
    │     → bunx @lythos/​skill-arena vs --config arena.toml
    │     → Each side spawns its player CLI process
    │
    └── Same player, different DECKS? (DEFAULT)
          → Agent-orchestrated — NO CLI
          → YOU spawn subagents via Agent tool
          → CLI prepare-workdir + CLI archive + parallel dispatch
          → Judge subagent collects + scores

Default: Agent-Orchestrated (single & cross-deck vs)

This is how arena works 95% of the time. The agent and CLI operate as a two-way control transfer protocol. Agent delegates mechanical invariants to CLI. CLI hands control back via its exit paths (success → next step; error → fix command). Agent stays in its own main loop — the subagent pattern is container spawn, not external RPC.

flowchart TD
    A["🤖 Agent: parse request"] --> B{Cross-PLAYER?}
    B -->|Yes| C[🔧 CLI vs --config]
    B -->|No — DEFAULT| D["🤖 Agent → 🔧 CLI: prepare-workdir"]
    D -->|"✅ workdir ready"| E["🤖 Agent: spawn subagents"]
    E --> F["🤖 Subagents: execute + write artifacts"]
    F --> G["🤖 Agent: collect + spawn judge"]
    G --> H["🤖 Judge: score → report.md"]
    H --> I["🤖 Agent → 🔧 CLI: archive"]
    I -->|"✅ archived"| J["🤖 Agent → 🔧 CLI: deck link restore"]

The protocol in one line: Agent hands to CLI (prepare-workdir, archive, deck link). CLI exits with success (✅ workdir ready → spawn) or HATEOAS error (❌ missing --deck → here's the fix command → retry). Agent reads the exit, decides next move, continues. Three CLI exit points, three handoffs back to agent.

single — test one deck

🤖→🔧 prepare-workdir --out /​tmp/​arena-xxx --brief "task"
    CLI exits: ✅ Workdir ready → 🤖 spawn subagent
🤖 Agent tool spawn: subagent executes in workdir, writes artifacts + decision-log-<cell-id>.jsonl
🤖→🔧 archive --from /​tmp/​arena-xxx --to ./​playground --sides side-a
    CLI exits: ✅ Archive complete → 🤖 done
🤖→🔧 deck link parent deck (restore)

cross-deck vs — compare decks A vs B

🤖→🔧 prepare-workdir × N (each side isolated, each with own deck)
    CLI exits: ✅ Workdir ready × N → 🤖 spawn N subagents in parallel
🤖 Agent tool spawn ×N, run_in_background=true
🤖 Collect: merge each side's per-cell decision logs → {side}/​decision-log.jsonl
🤖 Spawn judge subagent: score per criteria → report.md
🤖→🔧 archive --from /​tmp/​arena-xxx --to ./​playground --sides side-a,side-b
🤖→🔧 deck link parent deck (restore)

Why agent-orchestrated is default: Subagent = container spawn, not external RPC. Agent stays in its own main loop — can read subagent output, fix failures mid-run (switch mirror, adjust timeout, retry), spawn judge. Decision-log.jsonl from each subagent provides full observability. Cross-deck vs IS map-reduce — same agent type, different decks, parallel spawn, judge reduce.

Cross-Player Mode (OPT-IN, CLI only)

Use ONLY when comparing different players (kimi vs codex vs deepseek vs claude). The Agent tool can only spawn the same agent type — it CANNOT simulate another CLI's memory, hooks, or tool-use semantics. This is a hard runtime boundary, not a preference.

# Single deck, explicit player
bunx @lythos/​[email protected] single \
  --deck ./​skill-deck.toml \
  --brief "Investigate this repo" \
  --player kimi

# vs mode with arena.toml (each side's player in config)
bunx @lythos/​[email protected] vs --config ./​arena.toml

See references/​player-setup.md for player discovery, installation, and API key setup.

Standard Posture: Arena as Mindset Validator

Purpose: Verify that a skill's mental model (SOP, behavior pattern, decision chain) actually shapes agent behavior — not just that the skill file is read.

Minimal deck principle: Include ONLY the governance skill (lythoskill-deck) and the target skill under test. Extra skills dilute the signal — you are testing whether the target skill's intent survives when no other skills are there to compensate.

Standard posture (4 steps):

1. Prepare — prepare-workdir with minimal deck

   bunx @lythos/​[email protected] prepare-workdir \
     --deck ./​test-deck.toml \
     --out /​tmp/​arena-$(date +%Y%m%d-%H%M%S) \
     --brief "Execute the target skill's core workflow"
   

2. Dispatch — spawn subagent with decision-log mandate

   • Prompt MUST include: "Your working directory is {workDir}. Deck: {deckPath}. Task: {brief}. MANDATORY: write decision-log-{cellId}.jsonl to your CWD. Each line records a decision you made and why." — one cell id per cell, never a shared name.

3. Observe — collect decision-logs, not just artifacts

   • Read every decision-log-*.jsonl from workdir, or the merged decision-log.jsonl after the collect step
   • Check: did the subagent follow the skill's declared SOP?
   • Check: did the subagent stop at decision points and ask, or did it guess?
   • Check: are the decisions traceable to the skill's instructions?

4. Judge — score mindset alignment, not output correctness

   • "Did the subagent behave as the skill intended?" > "Was the output correct?"
   • A correct output achieved by guessing is a FAIL — the skill's mental model did not transfer.
   • A wrong output achieved by faithfully following the skill's SOP is valuable feedback — the skill's instructions need refinement.

Why this matters: A skill that declares "MUST FILL" but agents consistently leave empty has a mindset gap — the skill's intent is stated but not enforced by the agent's decision chain. Arena catches this before the skill reaches users.

Agent-Orchestrated Protocol

1. Setup — isolate per side

For EACH side, use prepare-workdir (same behavior as CLI single mode):

# Plan-first: review before executing
bunx @lythos/​[email protected] prepare-workdir \
  --deck ./​side-a.toml \
  --out /​tmp/​arena-$(date +%Y%m%d-%H%M%S)-side-a \
  --brief "task description" \
  --dry-run

# Execute (same command minus --dry-run)
bunx @lythos/​[email protected] prepare-workdir \
  --deck ./​side-a.toml \
  --out /​tmp/​arena-$(date +%Y%m%d-%H%M%S)-side-a \
  --brief "task description"

/​tmp is the experiment sandbox. Never run experiments in committed directories. Plan-first (--dry-run) shows skills, workdir path, link needed — review before IO.

2. Preflight self-check (BEFORE dispatch)

pwd && ls .claude/​skills/ 2>/​dev/​null || ls .agents/​skills/ 2>/​dev/​null && touch .arena-write-test && rm .arena-write-test && echo "OK"

If ANY fail → fix before proceeding.

3. Dispatch — parallel spawn

One subagent per cell:

subagent prompt:
  "You are an arena cell. Your working directory: {workDir}.
   Deck: {deckPath}.
   Task: {brief}
   MANDATORY: write decision-log-{cellId}.jsonl to your CWD.
   Each line: {"t":<seconds>,"phase":"...","decision":"...","reason":"..."}"

All subagents run in PARALLEL. Each writes to its own isolated workdir. No file conflicts.

Cell id is mandatory when cells share a workdir. Several cells of one side may run against the same workdir — a filename is a path, so two cells mandated to write the same decision-log.jsonl means the last writer erases the others (2026-09-09 run: side-a kept S1a's 9 entries, S3a/​S4a/​S5a gone). Give every cell a distinct id (s1a, s3a, …) and a per-cell name: the files cannot collide, and the collect step below merges them. A cell alone in its workdir may keep the bare decision-log.jsonl.

Platform note: run_in_background (or your platform's async spawn equivalent) keeps parent unblocked. Subagent inherits parent CWD — include "Your working directory is {workDir}" in the prompt so it cd's to the right place. Subagent skills load from the working set directory in that workdir (default .claude/​skills/).

4. Collect + Judge + Report + Archive

After ALL complete:

1. Collect

• Gather artifacts from each side's workdir
• Decision logs: each cell wrote decision-log-<cell-id>.jsonl. Read them per cell (they are the attributable evidence) or let archive merge them into one decision-log.jsonl per side — see the Archive step below. The merge is lossless and ordered by cell id, but a merged line carries no cell attribution, so the per-cell files are kept.

2. Judge

• Spawn judge subagent with all artifacts as context
• Score per criteria → write report.md

3. Archive (same behavior as CLI --out)

Use archive command (same copy logic as CLI single mode). Plan-first: dry-run to review what will be copied, then execute. archive is also the collect step for decision logs: when a side holds per-cell logs (decision-log-<cell-id>.jsonl), it merges them into <out>/<side>/​decision-log.jsonl and leaves the per-cell files in place. A side whose only log is the bare decision-log.jsonl is already merged and is copied untouched.

# Plan-first
bunx @lythos/​[email protected] archive \
  --from /​tmp/​arena-$(date +%Y%m%d-%H%M%S) \
  --to playground/​arena-$(date +%Y%m%d-%H%M%S) \
  --sides side-a,side-b \
  --report ./​report.md \
  --dry-run

# Execute (same minus --dry-run)
bunx @lythos/​[email protected] archive \
  --from /​tmp/​arena-$(date +%Y%m%d-%H%M%S) \
  --to playground/​arena-$(date +%Y%m%d-%H%M%S) \
  --sides side-a,side-b \
  --report ./​report.md

Archive contract (same skipSet as CLI --out: skips .claude, skill-deck.toml, skill-deck.lock, AGENTS.md) (same as CLI default):

4. Restore

• deck link --deck ./​skill-deck.toml

Reference passing (don't inline large context)

If task context is large (cortex cards, research notes), pass file REFERENCES, not inline text:

TASK: Review the API design.
Read: docs/​adr/​ADR-xxx.md, docs/​patterns/​xxx.md
Then implement in src/​.

Subagent has the same Read capability — shorter prompt, lower cost, can re-read. Use inlining only for small, self-contained tasks.

CLI Quick Reference

# single — most common
bunx @lythos/​[email protected] single \
  --deck ./​deck.toml --brief "task" --out ./​output

# vs — declarative config
bunx @lythos/​[email protected] vs --config ./​arena.toml

# Parameters
# --brief "<prompt>"    Inline task (primary input for single)
# --deck <path|url>     Deck for single subagent (URL auto-fetched)
# --player <name>       Only for cross-player: kimi|codex|deepseek|claude
# --timeout <ms>        Complex tasks need 300000-600000
# --out <dir>           All artifacts copy here after run
# --config <path>       arena.toml for vs mode
# --dry-run             Print execution plan without running

Constraints

• max 5 sides per arena run
• RESTORE parent deck after every run: deck link --deck ./​skill-deck.toml
• deny-by-default: skills not in the arena deck are invisible to subagents

Gotchas

CLI scaffolds, agent executes: The CLI only creates directories + deck files. It does NOT dispatch subagents or score outputs.

Agent tool CANNOT cross-player: Only Bun.spawn can call different CLI binaries. Agent tool spawn is same-agent only.

Judge is not a script: Semantic comparison ("which better fits the scenario") requires LLM inference. Token counting is scriptable; judgment is not.

vs does not pick a winner: Pareto frontier analysis — a cheap-medium-quality deck and expensive-high-quality deck can both be non-dominated.

Subagent spawn parameters (Claude Code baseline — adapt to your platform):

Supporting References