Can AI Generate Good Test Cases? A Practical Look

"Can AI generate test cases" is two questions wearing one search phrase. The first asks whether a language model can read a requirements document and produce the list of test cases a person would otherwise have typed. The second asks whether an agent can look at a running product and a specific code change, decide what needs checking, and go check it. The answers, the costs, and the failure modes are different. We build a product that does the second thing, so we will say where our interest lies and try to be fair to the first.

Two things people mean by generated test cases

Document-to-cases generation takes a spec, a user story, or a PRD and emits test cases as text: title, preconditions, steps, expected result. It is cheap and quick, and usually generic, because the model has only the document and the document rarely says how the product actually behaves. The output needs review before it is worth anything, and it produces no evidence, because nothing was executed.

Product-and-diff-to-plan generation is a different activity. The input is a pull request and a deployed preview of it. The output is an ordered plan of steps for that change, executed against the preview, with a verdict and artifacts. In TesterArmy this is the exploration agent, and the rest of this post uses its documented behaviour as the concrete case.

What "good" means for a generated test case

A generated case is good when it has four properties, and the first three are checkable without opinion.

It is specific to the change. "Verify login works" is true of every PR and therefore informative about none of them.

It is executable as written. In our guide to writing test steps, each step has one job, actions are split from assertions, and elements are named by the labels a user can see rather than by selectors or component names. A generated step that a teammate could not follow from the text alone is a step an agent will struggle with too.

It produces evidence. A case in a spreadsheet records what someone intended to check. A case that ran leaves a recording, screenshots, and a pass or fail per step.

It does not duplicate what your saved regression tests already cover. This one is a judgment call, and it is the property that still needs a human.

Before and after: the same PR, two kinds of output

Suppose a pull request renames the "Promo code" field at checkout to "Coupon", moves the discount to its own line in the order summary, and changes the validation for an empty code.

A document-fed generator, given the PR title and the checkout spec, tends to produce this:

Verify the checkout page loads.
Verify a user can apply a promo code.
Verify an invalid promo code shows an error.
Verify the order total is correct.

Every line is plausible and none of them is about this PR. Nothing mentions the rename, the new line, or the empty-code case the diff actually changed.

A plan built from the diff and the running preview reads differently:

Log in with the saved shopper account.
Add the Field Backpack to the cart from the product page.
Open the cart and verify the code field is labelled "Coupon".
Enter the saved test coupon in the Coupon field and apply it.
Verify the order summary shows a separate discount line and a reduced total.
Clear the Coupon field, apply it empty, and verify a validation message appears under the field.

The second plan is executable, tied to the three things the PR changed, and every step ends in something a screenshot can show. That is the whole gap between the two kinds of generation.

How the exploration agent builds a plan

The documented behaviour is worth quoting closely, because a generic description would oversell it.

For every pull request the agent works in two phases. In the plan phase it reads the PR, decides what is worth testing, and writes an ordered list of steps tailored to that change. In the execute phase it runs those steps in a real browser or mobile simulator and reports the result, the same way a saved test does. The plan considers the PR title and description, the changed files, and application context: project memory, test accounts, and uploaded files. When the project has a connected GitHub repository the agent reads the actual diffs rather than only file names. It covers functional checks, visual verification, and edge cases relevant to the change.

Before anything executes, a skip judge reviews the PR's diffs. If it decides the PR has no user-visible effect to test, a docs-only or config-only change for example, the run is skipped and the PR check reports a neutral "Tests skipped" status with the reason. Once the judge decides to test, the run always executes. On mobile projects the judge decides per platform, so an iOS-only change still tests on iOS while the Android run is skipped.

Results appear where the review is happening. The PR comment first shows the planned steps as a table, then updates in place with completed/total counts and per-step results as the run finishes. Regression groups and exploration runs keep separate comments, because they report different kinds of coverage.

Two knobs steer the planner. Project-level custom agent instructions, up to 2000 characters, are injected before it writes the plan; the docs give examples such as "Always sign in with the demo account first." and "Don't test the marketing site, only the app under /dashboard." They are advisory: the agent follows them unless doing so would prevent it from verifying the PR's actual changes. The other knob is the PR description, which the planner reads in full, up to GitHub's 65,536-character body limit, so instructions at the bottom of a long description count as much as the summary at the top. The docs recommend a ## TesterArmy testing instructions section at the end of every PR description: the entry route, numbered steps by visible label, expected behaviour after each action, the test account by its label rather than by credentials, required setup such as feature flags or seed data, and an out-of-scope line. If a coding agent writes your PRs, that convention goes into AGENTS.md once and every PR arrives planner-ready; the workflow around that is in using Claude Code and Codex to create and run e2e tests.

One boundary is deliberate: skip decisions come from the changed files, so a description cannot force the planner to skip a PR. And without a GitHub connection, on GitHub Enterprise or another CI, the agent can still be triggered through POST /api/v1/projects/{projectId}/pull-request-runs, but it then plans from the PR title and body, project memory, target URL, and saved credentials, without reading repository changes or posting GitHub results.

Where generation still needs a person

Agents do not know your business. The planner reads a diff and infers what a user would notice; it cannot know that the refund path is under regulatory scrutiny this quarter or that the coupon rename is cosmetic while the validation change is the risky part. That prioritisation stays with you, and the honest version of the argument is in autonomous QA: can an agent replace your test suite.

Deduplication is also yours. The docs describe saved regression groups and the exploration agent as running side by side and answering different questions; nothing in them says the planner removes steps a regression group already covers. In practice you scope the two: the regression group owns the flows that must be verified the same way on every PR, and custom instructions steer the exploration agent away from them or toward the areas you want probed. The plain-English regression tests are still written by people, and the craft is in how to write test cases in plain English.

Generated plans also inherit the product's determinism. When a change touches a feature whose output varies, a chat response or a generated summary, the plan has to assert on structure and behaviour rather than exact text, which is the subject of how to test AI apps. The planner will write such steps; whether they check the right thing is a review you do by reading the plan table in the PR comment.

So the practical answer is yes, with a condition. AI generates good test cases when it can see the product and the change, when the output is executed rather than filed, and when a person still decides what matters. Text generated from a document alone is a starting draft. Where either kind sits relative to your unit and integration tests is the subject of is the test pyramid still relevant in the age of AI agents.

FAQ

Can a language model generate test cases from a requirements document? Yes, and the output is usually a reasonable draft of generic cases: happy path, obvious validation, obvious error states. It cannot know how the product actually behaves or what changed in a given release, so every case needs review and none of them comes with evidence until someone runs it.

What is the difference between AI test case generation and an exploration agent? Generation from a document produces text for a person to execute or automate later. An exploration agent produces a plan from the pull request and the deployed preview, executes it in a real browser or simulator, and reports per-step results, screenshots, and a recording back to the PR, with a neutral skip when the diff has no user-visible effect.

Do generated test plans replace hand-written regression tests? No. TesterArmy's docs describe regression groups and the exploration agent as complementary: the groups verify known critical flows the same way on every PR, and the exploration agent adapts to what each PR changed. Most teams run both, with custom instructions keeping the exploration agent focused on what the regression group does not already cover.