A regression test must prove the bug can return

Start with the observable failure

A regression test should preserve the smallest public fact that made the bug harmful. Begin with the input, environment, and observable output from the report, then remove incidental details until the failure still occurs. If a date parser accepted an impossible date only when a timezone suffix was present, the contract is not that a particular helper gets called. It is that the public parser rejects that input in the documented way. This distinction matters when an agent has already seen the implementation: it can easily reproduce internal structure instead of capturing the behavior a user depends on.

Write the expected result before accepting the repair. Include the error type, status, state change, or returned value that callers can legitimately observe. Avoid asserting an entire object when two fields express the contract; broad snapshots create noise and make reviewers approve unrelated changes. This is practical reasoning, not a report that any test was executed for this article.

pytest parametrization ↗

Demand a red-to-green distinction

The strongest local evidence is a controlled sequence: the new test fails against the defective revision for the expected reason, then passes with the repair while adjacent tests remain green. Preserve the command, revision identifiers, and relevant failure excerpt. If reproducing the old revision is unsafe or impractical, say so and use a deliberately reverted line, a temporary fault, or a reviewed counterexample only when that method accurately represents the defect. A test added after the implementation that has never been observed failing may be useful, but it has not yet demonstrated that it detects this regression.

Do not confuse a failing setup with a detected bug. A fixture import error, unavailable service, or changed clock can produce red output while exercising none of the target behavior. Review the failure location and message, not only the exit code.

Protect valid alternatives

A narrow test can reject correct implementations; a wide test can demand behavior that the issue never specified. OpenAI's audits of SWE-bench task sets document both failure modes in benchmark tests. The lesson transfers to ordinary repositories: assertions need an explicit connection to requirements. Ask whether a different correct algorithm, data structure, or error path would still pass. If not, either the implementation detail is genuinely part of the contract and should be documented, or the assertion should be relaxed.

Pair the defect example with at least one neighboring valid case. For the hypothetical date parser, keep a valid leap-day example and a malformed input that fails for a different reason. Parametrization can make these boundaries visible without duplicating setup, but more rows do not automatically mean better coverage.

OpenAI audit of coding-evaluation tests ↗

Review the test as production code

Check determinism, fixture ownership, cleanup, names, and diagnostic quality. Time, randomness, locale, network access, and shared databases should be controlled at the closest useful boundary. A regression test that intermittently passes erodes trust in the whole suite. A test that needs a large integration environment may still be appropriate, but add the cheapest lower-level test that catches the same defect if the contract permits it.

Finish the handoff with four facts: the behavior protected, how the test was shown to fail, the commands actually run, and what remains untested. Do not claim more than the evidence shows. Passing the targeted test does not establish security, performance, or compatibility outside the exercised cases.