Review generated tests as claims about the product

Ask what each test claims

Translate every generated test into one sentence about observable behavior. If that sentence is merely the function returns what the function currently returns, the oracle may have been copied from the implementation. Trace the expected value to a requirement, public contract, bug report, standard, or reviewed domain rule. Tests can be syntactically correct and still institutionalize a mistake.

Group the suite by risk rather than file. Identify happy paths, boundaries, malformed input, authorization, state transitions, concurrency, and recovery. Delete duplicate cases that exercise the same branch without adding a new distinction. More test names and higher line coverage do not automatically create more confidence.

Break the implementation on purpose

Temporarily introduce a small representative fault or use a mutation-testing tool in an isolated branch: invert a comparison, remove validation, change a boundary, or suppress a state update. A meaningful test should fail for the intended reason. Surviving faults reveal weak assertions, unexercised paths, or equivalent changes that need human interpretation. Use mutation selectively on risk-heavy code because running many variants can be expensive.

This is stronger than watching the generated suite pass against the same implementation that inspired it. It also avoids assuming a test is good because it executes a line. No mutation run was performed for this article.

PIT mutation testing ↗

Inspect doubles and fixtures

Mocks should isolate a boundary, not replace the behavior under review. A test that asserts a mock was called with the implementation's current arguments can remain green while the user-visible result is wrong. Prefer assertions on returned state, durable effects, or an adapter contract. Keep at least one integration path for critical boundaries so the mock and real protocol cannot drift unnoticed.

Review fixture realism, isolation, cleanup, and mutability. pytest's parametrization documentation notes that parameter values are passed as-is, so mutable values changed by one case can affect another. Generated suites often hide shared state behind convenient fixtures. Run cases in different orders or independently when order dependence is plausible.

pytest parametrization behavior ↗

Audit the evaluator too

OpenAI's audits of SWE-bench variants found tests that were overly strict, under-covered requested behavior, or enforced underspecified requirements. The direct context is coding benchmarks, not every repository, but the review lesson is concrete: tests are fallible evaluators. Check that a valid alternative implementation passes and an incomplete implementation fails. Review skipped cases, broad snapshots, permissive exception assertions, and assertions that can never execute.

The handoff should name the protected risks, demonstrate selected tests failing under a relevant fault, list integration gaps, and estimate maintenance burden. Human reviewers own the product claims encoded by the suite. An agent can multiply examples quickly; it cannot supply missing requirements by repetition. Review test deletions and rewrites as carefully as additions, especially when a generated patch makes a previously failing check disappear. Require a written reason for weakened assertions, new skips, broader exception catches, or regenerated snapshots.

OpenAI SWE-bench Pro audit ↗