Evaluate an agent on your repository

Define the decision, not a leaderboard

Begin with the operational choice the evaluation must inform: whether an agent may draft tests, handle routine dependency updates, investigate incidents, or prepare reviewable feature slices. A single success percentage cannot answer all four. Public benchmarks help describe task families, but repository adoption depends on local architecture, documentation, build reliability, and review standards. SWE-bench evaluates attempts to resolve real repository issues in controlled environments; METR's evaluation resources likewise emphasize explicit tasks, environments, agents, and scoring. Borrow that structure without claiming comparability. Write a task contract, freeze the repository revision and tool configuration, and state what an accepted result means before running the agent. The output is evidence about one workflow under recorded conditions.

Sample work and non-work

Choose tasks from several risk and ambiguity bands. Include a localized bug with a known regression test, a change that spans two components, a diagnosis where no edit is needed, and a request that lacks enough information and should trigger clarification. Use completed historical tasks only if their solution is hidden from the agent and the current code can still reproduce the setup; otherwise create maintainers' exercises that resemble ordinary work. Include negative assertions such as protected files that must remain untouched. A task set containing only clean, solvable tickets measures persistence on favorable examples, not judgment. Record exclusions. If database migrations or production access are outside the proposed use, do not inflate the trial with them merely to appear comprehensive.

Worked example: a hypothetical six-task trial

Consider a hypothetical service evaluating an agent for low-risk maintenance. Six tasks are frozen at named commits: two parser bugs, one documentation synchronization, one dependency update, one flaky-test diagnosis, and one underspecified feature request. Two maintainers independently define acceptance tests and prohibited paths before any run. Each task receives the same time, tool, network, and retry limits. Reviewers see anonymized patches mixed with ordinary human maintenance patches and score behavioral correctness, scope discipline, maintainability, and evidence quality. The underspecified request passes only if the agent stops and asks for the missing policy. Results are reported per task, including invalid runs and reviewer disagreement. This does not estimate all future performance; it tests whether a bounded maintenance role is defensible.

Measure the cost of acceptance

For every run, retain the initial contract, configuration, trajectory or command record where policy permits, final diff, checks, and reviewer disposition. Track preparation time, agent elapsed time, reviewer time, correction rounds, and later defects found during a defined follow-up. Do not silently replace failed attempts with the best retry. Report pass-at-one separately from outcomes after human repair. Inspect contamination risks: repository tests may reveal the expected answer, generated tests may merely encode the patch, and graders may reject valid alternatives. A small, transparent evaluation with failure notes is more useful than a large opaque score. Repeat after material changes to the model, scaffold, permissions, or repository because the evaluated system is the complete workflow, not the model name alone.

SWE-bench evaluation guide ↗

METR autonomy evaluation resources ↗

Make aggregation auditable

Publish task-level evidence internally, not only an average. A decision maker should be able to see whether one spectacular success, one unusual failure, or a disputed grader dominates the result. Keep invalid runs in the denominator or explain their exclusion, and predefine how ties and reviewer disagreement are resolved.