TesterArmyTesterArmy
TesterArmyTesterArmy
WelcomeIntroductionQuick Start
Run ModePull Request TestingProduction MonitoringGroup Webhooks
Test CredentialsAgent Mail InboxesHTTP Basic Auth
OverviewApp UploadsGitHub Actions
VercelCoolifyCoolifyCustom InfrastructureSlack
Testing Staging Environment
Getting StartedLocal Development
Run

Pull Request Testing

Pull Request Testing runs tests against every preview deployment and posts results as a GitHub comment. You can run two kinds of tests on each PR, and they work side by side:

  • Test group runs - a predefined test group runs against the preview URL. Great for catching regressions in existing application flows.

    Predefined tests PR comment

  • Exploration agent - an AI agent reads the PR context, generates a targeted test plan, and executes it in a real browser. Great for surfacing issues nobody thought to write a test for yet.

    Exploration agent PR comment

Prerequisites

Run Mode

Previous Page

Production Monitoring

Next Page

On this page

PrerequisitesSupported providersTest group runsSetupExploration agentHow it worksTroubleshootingNo comment appears on the PRTests run against the production URL
  • GitHub App installed and connected to the repository.
  • A project linked to a GitHub repository.
  • Deployment provider configured - see Vercel, Coolify, or Custom infrastructure setup guides.

Supported providers

  • Vercel - automatic via GitHub deployment status events. No webhook needed.
  • Coolify - via webhook configured in Coolify notifications.
  • Custom infrastructure - via group webhooks, triggered from your own CI (GitHub Actions, GitLab CI, Jenkins, shell scripts, etc.).

Test group runs

Reviewers see pass/fail results directly in the PR without leaving GitHub, and broken changes get caught before merge.

Setup

  1. Open your project and go to the PR Testing tab.
  2. Enable Test groups for PR runs and select the test group to run on pull requests. All tests in this group will execute against every new deployment preview.
  3. If Vercel deployment protection is enabled, add a bypass token.

Every new preview deployment now triggers the selected test group automatically.

Exploration agent

The plan is tailored to each PR and considers:

  • PR title and description - understands intent.
  • Changed files - focuses on affected areas.
  • Application context - project memory, test accounts, uploaded files.

It covers functional checks, visual verification, and edge cases relevant to the changes. No configuration is needed beyond connecting the GitHub App and a deployment provider - the agent runs automatically on each new preview deployment.

How it works

  1. A pull request is opened or updated.
  2. The deployment provider creates a preview and signals TesterArmy (via GitHub deployment status events for Vercel, or a webhook for Coolify).
  3. TesterArmy resolves the PR from the commit SHA and triggers the configured test group and/or the exploration agent against the preview URL.
  4. Results are posted as a single comment on the PR.
  5. On subsequent pushes, the same comment is updated in place rather than creating a new one.

Troubleshooting

No comment appears on the PR

  1. GitHub App not connected - verify installation in Project Settings.
  2. Permissions - the GitHub App needs Pull requests: Read & Write.
  3. No tests configured - if you're relying on test groups, make sure one is enabled in the PR Testing tab.

Tests run against the production URL

  1. Deployment not detected - TesterArmy only triggers on preview deployments. Verify the deployment shows up in your provider's dashboard.
  2. Bypass token missing - if the preview URL returns an auth screen, add a bypass token.