How Automated PR Testing Speeds Up Code Review

A reviewer opening a pull request today has four ways to answer the question "does this actually work." They can read the diff and reason about it. They can pull the branch and click through the feature locally. They can open the preview deployment and click through it there. Or they can trust the author, who probably clicked through it once before pushing. Each option is paid for in the reviewer's attention, and the last one is paid for later by the team. This post is about what changes when every PR arrives with its own evidence. We build the agent that supplies it, so read with that in mind.

What a code review is for

Strip a review down and it has two jobs that get blended. One is judgment: is this the right approach, does it fit the architecture, will the next person understand it. The other is verification: when this branch is deployed, does the flow it touches still work for a user. The first needs a senior human. The second needs a browser, a test account, and patience, and it pulls the reviewer out of the diff and into a login screen.

Automated PR testing separates the two. The agent does the clicking on the preview deployment and leaves the result on the PR. The reviewer reads the verdict, opens the recording when it is red or when the change is visual, and spends the rest of the review on judgment.

What arrives with the pull request

With TesterArmy connected, a pull request that produces a preview deployment gets two kinds of coverage, and they work side by side.

Regression groups are the tests you saved: plain-English flows for the paths you already know matter, run the same way against every preview. Each selected group runs as its own batch with its own GitHub check and its own PR comment.

The exploration agent has no saved steps. It reads the pull request, writes a test plan tailored to that change, and executes it in a real browser or mobile simulator. Its comment first shows the planned steps as a table, then updates in place with completed and total counts and per-step results as the run finishes. On subsequent pushes each surface updates its existing comment rather than adding a new one. How the planner decides what to test is covered in can AI generate good test cases.

Behind each check is the run itself: a recording, screenshots from the important points of the run, a result per step, and reported issues explaining the product behaviour that stopped a step from passing. Start with the first failed step; later failures are often consequences of the same blocker.

Ethan Shea, Co-Founder and CTO at Regbase, put the effect this way on our homepage: "TesterArmy is a core part of my software factory. Exploration tests for every PR is a killer feature."

How it is wired

The mechanics are documented in pull request testing and the GitHub integration; the short version follows.

You install the TesterArmy GitHub App, grant it access to the repositories it should test, and connect one GitHub.com repository to a TesterArmy project; a connection cannot be shared by multiple projects. Optionally you pick a target branch, main for example, so only PRs aimed at that base branch trigger automatic runs.

Then you connect preview delivery. For Vercel, TesterArmy is triggered by GitHub deployment status events and verifies that the deployment URL belongs to the Vercel project you selected. Netlify and Coolify send a deploy notification webhook, and any other CI calls a group webhook or the API after deployment. If previews are protected you add a bypass token; the Vercel walkthrough is in running e2e tests on Vercel preview deployments.

In the project's PR Testing tab you enable "Test groups for PR runs" and select the groups, enable the exploration agent, or both. The exploration agent needs no step authoring; to steer it, custom agent instructions of up to 2000 characters are injected into the planner, and a ## TesterArmy testing instructions section at the end of the PR description is read in full.

"Delay automatic test runs" adds a quiet period: TesterArmy waits until the PR has had no new deployments for the configured number of minutes, then tests the latest commit once. It is on by default at 3 minutes; the first deployment of a PR is never delayed.

Red means the product broke

Whether reviewers keep trusting the checks depends on what a red result is allowed to mean. Older end-to-end setups taught reviewers that red usually meant the test environment, so red got ignored. TesterArmy's results model separates whether execution finished from whether the tested behaviour worked. A completed run ends in one of three outcomes. PASSED means all required behaviour passed. FAILED means execution finished and found a product, configuration, or test-step issue. BLOCKED means an environment or setup problem, or an agent automation limit, stopped the test before a product verdict was possible. Blocked runs carry an output.blockedReason describing what to fix: a down environment, missing or rejected credentials, missing seed data, or an agent that ran out of actions or time on a step. They are excluded from failure counts and pass rates, and they conclude the GitHub check as neutral.

For a reviewer this is the whole point. When the check is red, someone shipped a product problem and the evidence is attached. When the preview never came up, or the login step could not get a credential, or a bot challenge stopped the agent, the check says so and nobody debates whether the code is at fault. Two common blocked causes have their own guides: authentication that needs a one-time code, in how to test OTP and 2FA flows automatically, and previews behind a challenge, in how to test flows behind a CAPTCHA.

Making it a gate, carefully

By default TesterArmy checks are advisory. A failed test is reported in the check title and summary, but the check concludes neutral, and GitHub counts neutral as passing, so a failed test never blocks a merge until you ask it to. Turn on "Block merges on failed tests" in the PR Testing tab and failed tests conclude as failure; then mark the checks you care about as required in branch protection or a ruleset. The check names are TesterArmy / Exploration Test and TesterArmy / <group name>; when several projects share a repository, each project's checks carry a (<project name>) suffix.

Three caveats from the docs are worth repeating. Runs without a product verdict stay neutral even with blocking on: blocked runs, runs that crashed before a verdict, cancellations, and PRs the exploration agent skipped. A check run only exists after a deployment TesterArmy accepted, so a required check on a PR that never produces that deployment waits indefinitely. And while a delay is pending, the newest commit has no TesterArmy check yet.

The pattern that works: run advisory first, watch the checks agree with reality across a stretch of PRs, then promote the groups you trust to required, and leave the exploration check advisory a little longer, since its plan changes with every PR.

What changes in the review conversation

The reviewer now reads a table of steps and a verdict before the diff. If the exploration agent decided the PR had no user-visible effect, a docs-only or config-only change for instance, the check says "Tests skipped" with the reason, which tells the reviewer there was nothing to click through.

The author learns about a failure earlier. The failing step, its screenshot, and the recording land on the PR when the preview deploys, usually before a reviewer has been assigned, so authors fix their own regressions without a round trip.

The same machinery covers mobile pull requests, where the exploration agent runs on a simulator or emulator and the skip judge decides per platform; teams coming from native frameworks will find the trade-offs in why teams are replacing Detox for React Native testing. What none of this covers is judgment. Whether the feature is confusing and whether the approach is right remain human work, whether that human is a reviewer on the team or an outsourced QA function. Where PR-level journey checks sit relative to unit and integration tests is answered in is the test pyramid still relevant in the age of AI agents.

FAQ

Does automated PR testing require a preview deployment? Yes. TesterArmy's hosted runners test a deployed URL, so the PR needs a preview from Vercel, Netlify, Coolify, or your own CI calling a webhook or the API after deployment. Automatic runs trigger only on preview deployments, never on the production URL.

Can automated PR tests block a merge? By default the checks are advisory and conclude neutral, which GitHub counts as passing. Enable "Block merges on failed tests" in the project's PR Testing tab, then mark TesterArmy / Exploration Test or TesterArmy / <group name> as required status checks in branch protection. Blocked, cancelled, crashed, and skipped runs stay neutral even with the setting on.

What happens when the preview environment is down during a PR run? The run completes as BLOCKED with an output.blockedReason naming the cause, the GitHub check concludes neutral, and the run is excluded from failure counts and pass rates. Restore the environment and retry; the product is not marked as failing for an infrastructure problem.