Introducing Scout: API Testing Built for AI Agents

Coding agents are great at writing API code and surprisingly bad at testing it. Ask an agent to "check if my API works" and it will happily curl random endpoints, follow redirects to hosts it should never touch, paste your bearer token into its output, and - if you're unlucky - send a DELETE to production.

The problem isn't the agent's judgment. It's that raw HTTP tools have no guardrails. So we built Scout: an open source, OpenAPI-driven API testing CLI designed to be operated by agents. Scout parses your spec, executes instrumented requests, validates responses against schemas, and records structured findings - while locking requests to a single host, blocking writes by default, rate limiting, enforcing a request budget, and redacting credentials.

In this post we will walk through testing an API with Scout from scratch. We'll use the Swagger Petstore as the target, and - spoiler - Scout finds three real bugs in it in under a minute.

Why a harness, not just curl

The core idea behind Scout: the agent is the operator, Scout is the harness. The agent decides what to test; Scout enforces how requests are allowed to happen:

• Host lock - requests only go to the configured base URL host.
• Mutation gate - POST, PUT, PATCH, and DELETE are blocked unless you explicitly opt in.
• Rate limit and budget - 5 requests/second and 300 requests per run by default, shared across every command.
• Secret redaction - credentials are resolved from environment variables at request time and scrubbed from output, so tokens never end up in your agent's context window.
• Bounded responses - response bodies are capped at 1 MiB previews, so a huge payload can't blow up the context.

Every request also comes back with a verdict: did the status match the spec, did the body validate against the schema, was the content type right, how long did it take. That structure is what makes the output useful to an agent instead of a wall of JSON to eyeball.

Getting started

Scout ships as an npm package, so there is nothing to install. Point init at an OpenAPI spec:

npx @testerarmy/scout init https://petstore3.swagger.io/api/v3/openapi.json

Scout downloads and caches the spec, reads the base URL from it, and starts a run:

{
  "spec": { "title": "Swagger Petstore - OpenAPI 3.0", "version": "1.0.27" },
  "operations": 19,
  "baseUrl": "https://petstore3.swagger.io/api/v3",
  "policy": { "allowMutations": false }
}

If your API needs auth, pass headers as environment variable references - never literal tokens:

npx @testerarmy/scout init openapi.json \
  --header 'Authorization: Bearer $API_TOKEN'

No spec? Scout also runs spec-less with init --base-url https://api.example.com. You lose schema validation and coverage, but the host lock, mutation gate, and budget still apply.

💡 All commands support --json, and non-interactive output defaults to JSON - which is exactly what you want when an agent is driving.

Exploring the API

Before sending traffic, let's see what we're working with. endpoints searches the operation index:

npx @testerarmy/scout endpoints --tag pet

{
  "total": 8,
  "operations": [
    { "method": "GET", "path": "/pet/{petId}", "summary": "Find pet by ID." },
    {
      "method": "POST",
      "path": "/pet",
      "summary": "Add a new pet to the store."
    },
    {
      "method": "GET",
      "path": "/pet/findByStatus",
      "summary": "Finds Pets by status."
    }
  ]
}

And schema shows one operation's full contract - parameters, request body, and response schemas - without the agent having to parse the raw spec.

Running a baseline sweep

sweep plans a read-oriented baseline across the whole API: happy paths, required-query probes, not-found shapes, and missing/invalid auth checks. Always start with --dry-run to review the plan before any traffic:

npx @testerarmy/scout sweep --dry-run --max-requests 10

{
  "probesPlanned": 10,
  "plan": [
    {
      "operation": "GET /pet/{petId}",
      "kind": "not-found-shape",
      "disposition": "planned"
    },
    {
      "operation": "GET /store/inventory",
      "kind": "happy-path",
      "disposition": "planned"
    },
    {
      "operation": "GET /store/inventory",
      "kind": "missing-auth",
      "disposition": "planned"
    },
    {
      "operation": "POST /pet",
      "disposition": "ineligible",
      "reason": "unsafe-method"
    }
  ]
}

Note the POST operations marked ineligible: unsafe-method - mutations stay off unless you opt in. The plan looks reasonable, so let's run it for real:

npx @testerarmy/scout sweep --max-requests 10

Ten requests later, Scout comes back with three findings against the live Petstore:

{
  "probesRun": 10,
  "findingsCreated": 3,
  "findings": [
    {
      "severity": "high",
      "endpoint": "GET /store/inventory",
      "title": "Server error 500 on happy path",
      "evidence": ["GET .../store/inventory -> 500 (208ms)"]
    },
    {
      "severity": "medium",
      "endpoint": "GET /store/order/{orderId}",
      "title": "Server error 500 for missing resource",
      "description": "A lookup with a synthetic id produced a 5xx instead of a clean 404."
    },
    {
      "severity": "medium",
      "endpoint": "GET /user/{username}",
      "title": "Server error 500 for missing resource"
    }
  ]
}

These are real bugs. GET /store/inventory - a parameter-free happy path - returns a 500, and two lookup endpoints throw 500s instead of clean 404s for missing resources. Every finding includes evidence and a repro command, so anyone (human or agent) can reproduce it with a single scout call.

Probing single endpoints

For targeted investigation, call sends one request and returns the response plus a verdict:

npx @testerarmy/scout call GET '/pet/{petId}' --path-param petId=1 --expect 200

{
  "response": { "status": 404, "body": "Pet not found" },
  "verdict": {
    "expectMatched": false,
    "statusExpected": true,
    "summary": "FAIL · expect: MISMATCH · status: expected · schema: n/a · 664ms"
  }
}

The verdict distinguishes "this status is documented in the spec" from "this is what I expected here" - useful signal when you're chasing a bug rather than validating the contract.

Calls can also chain. Capture a value from one response and reference it in the next:

scout call POST /users --data '{"name":"scout-test-ada"}' --capture uid=id
scout call GET /users/{id} --path-param 'id={{uid}}' --expect 200
scout call DELETE /users/{id} --path-param 'id={{uid}}' --expect 204
scout call GET /users/{id} --path-param 'id={{uid}}' --expect 404

That four-line create → read → delete → verify-gone loop is the bread and butter of lifecycle testing, and it's exactly the kind of flow an agent can run autonomously. Note this requires --allow-mutations at init time - and clearly fake test data.

Fuzzing request bodies

fuzz generates negative cases from the operation's schema: malformed JSON, nulls, wrong types, missing required fields, boundary values, oversized strings, and unknown fields. Preview the plan first:

npx @testerarmy/scout fuzz POST /pet --dry-run

{
  "operation": "POST /pet",
  "casesPlanned": 25,
  "cases": [
    {
      "id": "malformed-json",
      "title": "Malformed JSON body",
      "expectedReject": true
    },
    {
      "id": "missing-required-name",
      "title": "Missing required field /name",
      "expectedReject": true
    },
    {
      "id": "wrong-type-photoUrls",
      "title": "Wrong type at /photoUrls",
      "expectedReject": true
    },
    {
      "id": "oversized-name",
      "title": "Oversized string at /name",
      "expectedReject": false
    }
  ]
}

Twenty-five cases from a single schema, each tagged with whether the API should reject it. When a fuzz case gets a 2xx where it expected rejection, Scout records it as a candidate finding - you confirm or dismiss it after checking it's a real issue:

scout finding list
scout finding confirm <id>
scout finding dismiss <id>

Reports and CI gates

When you're done, report summarizes the run as JSON or Markdown:

npx @testerarmy/scout report --md report.md

# Scout report: Swagger Petstore - OpenAPI 3.0 `1.0.27`

**Result:** FAIL · **Run:** complete
**Findings:** 3 · **Coverage:** 8/19 operations (42%)

### 🟠 [high] Server error 500 on happy path

- **Endpoint:** `GET /store/inventory`
- **Repro:** `scout call GET /store/inventory`

The interesting part is --ci, which turns the report into a pipeline gate:

npx @testerarmy/scout report --ci --severity-threshold high --min-coverage 40

It exits non-zero when a confirmed finding meets the severity threshold or coverage falls short. Run Scout on every PR and a 500-on-happy-path never reaches production.

Handing Scout to your agent

Everything above works by hand, but Scout is built to be driven by an agent. One command installs a version-matched skill into your project:

npx @testerarmy/scout@latest agent init

Then prompt your agent - Claude Code, Codex, opencode, whichever you use:

Test my API for bugs with Scout. Exercise it through realistic user flows,
not by reviewing endpoints in isolation.

- Spec: https://api.example.com/openapi.json
- Auth: Authorization: Bearer $API_TOKEN

Ask before testing POST, PUT, PATCH, or DELETE. Investigate anomalies with
scout call and scout fuzz, record verified bugs with scout finding add,
then run scout report.

The skill teaches the agent the full workflow: orient with endpoints and schema, sweep, build valid controls before interpreting negative results, fuzz, validate findings before confirming them, and stop when scope gets ambiguous. The guardrails mean the worst-case blast radius is bounded - even if the agent gets creative.

Note: Guardrails do not replace authorization. Only point Scout at APIs you're allowed to test, and confirm environment, methods, and test data before enabling mutations.

That's a wrap

Ten requests against a public demo API produced three confirmed bugs with evidence and repro commands. That's the pitch: give your agent a harness instead of a footgun, and API testing becomes something you can run on every PR instead of something you do manually before a release.

Scout is open source and MIT licensed - try it with npx @testerarmy/scout init <your-spec>. And if you want the same agent-driven approach for your whole app - browser and mobile flows included - that's what we build at TesterArmy.