advanced-evaluation
If you're building LLM-as-judge systems, this gives you the production patterns that actually matter: position bias mitigation through...
harness-engineering
You need this when you're building agents that run for hours or days without supervision.
Full skill instructions
Harness engineering designs the control system around an agent: what it may edit, how it receives feedback, where it writes state, how failures recover, and who can approve irreversible actions. The harness is the difference between a helpful agent session and an autonomous loop that can run for days without corrupting its objective.
Activate this skill when:
Do not activate this skill for adjacent work owned by other skills:
evaluation.tool-design.project-development.hosted-agents.Separate the agent from the environment it operates inside. The agent proposes actions; the harness defines allowed surfaces, feedback, persistence, and promotion rules.
Use four surface classes:
| Surface | Examples | Rule |
|---|---|---|
| Locked | Eval metric, rubric, validation script, merge policy | Agent may read and propose changes, but cannot score itself with modified rules |
| Editable | Skill draft, experiment file, prompt, config under test | Agent may mutate during the loop |
| Append-only | Results log, research thread, rejected ideas | Agent may append, not rewrite |
| Human-controlled | Merge, production deploy, credentials, destructive operations | Requires explicit human approval |
Autonomy works when feedback is fast, unambiguous, and hard to game. Karpathy's autoresearch is the minimal pattern: one editable file, one locked evaluation file, fixed wall-clock budget, one scalar metric, git rollback, and a durable results log. The lesson is not that every harness needs one metric; it is that ambiguous feedback creates ambiguous autonomy.
For open-ended research-to-skill work, replace the scalar metric with locked rubrics, deterministic structure checks, source traceability, and human review thresholds.
Long-running agents must externalize state. Store plans, source queues, results, failures, and handoffs in files so future agents can resume without relying on chat history. Prime Intellect's autonomous nanoGPT work showed the value of durable scratchpads and THREAD.md-style logs for recovery, monitoring, and audit.
Use append-only logs for:
Agents tend to exploit the nearest surface, stack complexity, and under-run pruning. Add explicit search rules:
For research-to-skill systems, track accepted mechanisms separately from prose. A mechanism record should include a stable mechanism_id, owning_skill, status, activation scenario, behavior change, evidence, and failure modes. Novelty gates should compare against this registry before using broader corpus overlap, because keyword overlap catches stale phrasing while mechanism comparison catches real duplication.
Autonomous agents may prepare PRs, but governance must be explicit. They can draft changes, run checks, and write PR summaries. They should not merge, deploy, or push without human approval unless the user has explicitly granted that permission for the specific action.
Use this pattern when optimizing an artifact against a stable evaluator:
read locked context -> choose hypothesis -> edit allowed surface -> commit/checkpoint
-> run evaluator -> log result -> keep if better -> discard or rollback if worse
-> repeat
Required properties:
Use this pattern when sources become skill changes:
discover -> retrieve -> gate -> score -> extract mechanism
-> map to existing or new skill -> draft proposal -> validate structure
-> prepare PR -> human review
The locked evaluator is a combination of source rubrics, skill-change rubrics, structure checks, and reviewer approval. The editable artifact is the proposed skill delta.
Assume an optimizing agent will learn the harness. Guard against:
Mitigation: lock rubrics per run, report per-dimension scores, require source retrieval evidence, preserve rejected attempts, and route governance changes to human review.
Use monitoring agents for long runs, but restrict them to read-only reporting unless explicitly tasked otherwise. Monitoring output should report:
research-run/
THREAD.md
sources/
queue.md
evaluations/
proposals/
logs/
results.tsv
rejected.md
drafts/
Use TSV or JSONL for append-only machine-readable logs. Use Markdown for handoffs and reviewer-facing summaries.
Example 1: Locked metric
An agent optimizes train.py, but prepare.py owns data loading and evaluation. The agent can edit the model but cannot change the metric. Failed experiments are logged and rolled back.
Example 2: Locked rubric
An agent evaluates a new Anthropic or OpenAI engineering post, but the source curation rubric is locked for the run. If the source passes, the agent drafts a skill proposal. It cannot lower the rubric threshold to admit the source.
Example 3: Auto-PR without auto-merge
An agent prepares a branch and PR body after passing source, skill, and structure checks. The PR states unresolved risks and waits for human merge approval.
This skill connects to:
Internal references:
researcher/README.md - Read when implementing the repo-native research-to-skill operating systemresearcher/rubrics/harness-change.md - Read when evaluating changes to an agent harnessresearcher/runbooks/autonomous-research-loop.md - Read when running a source-to-skill loopExternal resources:
autoresearch - Constrained autonomous experiment loop with locked evaluationCreated: 2026-05-14 Last Updated: 2026-05-15 Author: Agent Skills for Context Engineering Contributors Version: 1.1.0
If you're building LLM-as-judge systems, this gives you the production patterns that actually matter: position bias mitigation through...
This handles the full lifecycle of testing agent systems: deterministic checks before you burn tokens on LLM judges, multi-dimensional...
>
This skill should be used when writing, enhancing, or evaluating the launch prompt for a long-running autonomous agent or a parallel multi-agent orchestration attacking a hard problem: pseudo-formal task briefs that define terms and an exact success predicate linguistically, enumerate non-counting outcomes, set persistence rules with explicit stop and return conditions and effort floors, manage a diverse portfolio of parallel approaches with an approach registry and blocked-route bookkeeping, and gate the return on adversarial audit. Route agent topology and coordination protocols to multi-agent-patterns, runtime control surfaces and loop governance to harness-engineering, evaluator and quality-gate construction to evaluation, judge design to advanced-evaluation, and compaction or memory mechanics to context-compression and memory-systems.
If you're running orchestrator-worker agent systems and watching token costs compound as the orchestrator's trajectory gets replayed to...
If you're building agents that run in the cloud instead of locally, this is your playbook.
Most agents hit context limits faster than you'd expect, and this skill treats the filesystem as overflow storage instead of cramming...
This is the conceptual foundation for the entire context engineering collection.
Before you write a single line of code for an LLM project, this walks you through whether an LLM is even the right choice for your task.
If you're building agents that need to explain their reasoning or coordinate mental states across a multi-agent system, this gives you...
This is the skill for designing the tool layer between deterministic code and non-deterministic agents.
Generate, edit, and beat-sync AI video with leading models in one workspace.
The world's fastest calendar for remote work
Transform Your Design with AI Designer by ImgCreator.ai
Revolutionizing Video Production with AI-Powered Creativity
Extend an image past the frame and let AI fill the new aspect ratio.
Discover your celebrity doppelgänger with StarByFace!
ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.
Opus.ai: Revolutionize Your Web Experience