Introducing Scout: API Testing Built for AI Agents

Coding agents are great at writing API code and surprisingly bad at testing it. Ask an agent to "check if my API works" and it will happily curl random endpoints, follow redirects to hosts it should never touch, paste your bearer token into its output, and - if you're unlucky - send a DELETE to production.

The problem isn't the agent's judgment. It's that raw HTTP tools have no guardrails. So we built Scout: an open source, OpenAPI-driven API testing CLI designed to be operated by agents. Scout parses your spec, executes instrumented requests, validates responses against schemas, and records structured findings - while locking requests to a single host, blocking writes by default, rate limiting, enforcing a request budget, and redacting credentials.

In this post we will walk through testing an API with Scout from scratch. We'll use the Swagger Petstore as the target, and - spoiler - Scout finds three real bugs in it in under a minute.

Why a harness, not just curl

The core idea behind Scout: the agent is the operator, Scout is the harness. The agent decides what to test; Scout enforces how requests are allowed to happen:

• Host lock - requests only go to the configured base URL host.
• Mutation gate - POST, PUT, PATCH, and DELETE are blocked unless you explicitly opt in.
• Rate limit and budget - 5 requests/second and 300 requests per run by default, shared across every command.
• Secret redaction - credentials are resolved from environment variables at request time and scrubbed from output, so tokens never end up in your agent's context window.
• Bounded responses - response bodies are capped at 1 MiB previews, so a huge payload can't blow up the context.

Every request also comes back with a verdict: did the status match the spec, did the body validate against the schema, was the content type right, how long did it take. That structure is what makes the output useful to an agent instead of a wall of JSON to eyeball.

Getting started

Scout ships as an npm package, so there is nothing to install. Point init at an OpenAPI spec:

Scout downloads and caches the spec, reads the base URL from it, and starts a run:

If your API needs auth, pass headers as environment variable references - never literal tokens:

No spec? Scout also runs spec-less with init --base-url https://api.example.com. You lose schema validation and coverage, but the host lock, mutation gate, and budget still apply.

💡 All commands support --json, and non-interactive output defaults to JSON - which is exactly what you want when an agent is driving.

Exploring the API

Before sending traffic, let's see what we're working with. endpoints searches the operation index:

And schema shows one operation's full contract - parameters, request body, and response schemas - without the agent having to parse the raw spec.

Running a baseline sweep

sweep plans a read-oriented baseline across the whole API: happy paths, required-query probes, not-found shapes, and missing/invalid auth checks. Always start with --dry-run to review the plan before any traffic:

Note the POST operations marked ineligible: unsafe-method - mutations stay off unless you opt in. The plan looks reasonable, so let's run it for real:

Ten requests later, Scout comes back with three findings against the live Petstore:

These are real bugs. GET /store/inventory - a parameter-free happy path - returns a 500, and two lookup endpoints throw 500s instead of clean 404s for missing resources. Every finding includes evidence and a repro command, so anyone (human or agent) can reproduce it with a single scout call.

Probing single endpoints

For targeted investigation, call sends one request and returns the response plus a verdict:

The verdict distinguishes "this status is documented in the spec" from "this is what I expected here" - useful signal when you're chasing a bug rather than validating the contract.

Calls can also chain. Capture a value from one response and reference it in the next:

That four-line create → read → delete → verify-gone loop is the bread and butter of lifecycle testing, and it's exactly the kind of flow an agent can run autonomously. Note this requires --allow-mutations at init time - and clearly fake test data.

Fuzzing request bodies

fuzz generates negative cases from the operation's schema: malformed JSON, nulls, wrong types, missing required fields, boundary values, oversized strings, and unknown fields. Preview the plan first:

Twenty-five cases from a single schema, each tagged with whether the API should reject it. When a fuzz case gets a 2xx where it expected rejection, Scout records it as a candidate finding - you confirm or dismiss it after checking it's a real issue:

Reports and CI gates

When you're done, report summarizes the run as JSON or Markdown:

The interesting part is --ci, which turns the report into a pipeline gate:

It exits non-zero when a confirmed finding meets the severity threshold or coverage falls short. Run Scout on every PR and a 500-on-happy-path never reaches production.

Handing Scout to your agent

Everything above works by hand, but Scout is built to be driven by an agent. One command installs a version-matched skill into your project:

Then prompt your agent - Claude Code, Codex, opencode, whichever you use:

The skill teaches the agent the full workflow: orient with endpoints and schema, sweep, build valid controls before interpreting negative results, fuzz, validate findings before confirming them, and stop when scope gets ambiguous. The guardrails mean the worst-case blast radius is bounded - even if the agent gets creative.

Note: Guardrails do not replace authorization. Only point Scout at APIs you're allowed to test, and confirm environment, methods, and test data before enabling mutations.

That's a wrap

Ten requests against a public demo API produced three confirmed bugs with evidence and repro commands. That's the pitch: give your agent a harness instead of a footgun, and API testing becomes something you can run on every PR instead of something you do manually before a release.

Scout is open source and MIT licensed - try it with npx @testerarmy/scout init <your-spec>. And if you want the same agent-driven approach for your whole app - browser and mobile flows included - that's what we build at TesterArmy.