How to Test AI Apps: Beyond Evals

Most teams building on a language model already run evals: a dataset of prompts, a scoring function, a dashboard that goes red when a model or prompt change lowers the score. Evals are necessary, and they answer a narrow question: does a given input produce an acceptable output from the model. The product your users touch is a different object: a login, a streaming response area, tool calls that fan out to your own services, a retrieval layer with links in it, and session state that has to survive a page refresh. Each of those can break while every eval stays green. We build TesterArmy, an agent that tests apps like real users, and a growing share of what it tests is AI products, so this is a guide from that vantage point, with the boundaries stated.

What evals measure, and the object they leave out

An eval isolates the model. It holds the prompt and the dataset fixed, varies the model or the system prompt, and scores the text that comes back. That is the right instrument for a model regression: a checkpoint that hallucinates more, a prompt edit that drops a formatting rule. It says nothing about the journey a user takes to reach that text, and an exception tracker only catches the part of the journey that throws. The seams between prompt, tool call, retrieval, and response are where AI apps break, and each piece can work alone while the chain fails as a whole. A journey test starts at the login page and ends at the outcome the user wanted, on a real deploy, which is the thing neither instrument is looking at.

Six failures that pass every eval

The first is a streaming interface that never renders: the API streams tokens correctly, the component rendering partial output throws on an unexpected chunk, and the model's good answer is never seen. The second is a tool call that fails silently: the assistant calls your booking service, receives an error, and still produces a courteous "I have booked that for you", which an eval on the final text may score as helpful. The third is a correct RAG answer with a dead citation, where the answer paraphrases the right source and the link behind it returns an error after a documentation reorganisation. The fourth is session state lost around authentication: the conversation resets when a user signs in mid-chat, or the uploaded file is no longer attached to the next turn. The fifth is latency that breaks the flow, a first token arriving after the user has left, with the eventual answer scoring perfectly. The sixth is a prompt change that shifts the interface: a system prompt edited to return Markdown tables when the renderer has no table support, so the eval reads clean text and the user sees raw pipes.

None of these is a model failure, and all are product failures a user feels immediately. Our AI app testing page frames the gap from the buyer's side; the rest of this post is about writing tests that catch it.

Writing journey tests when the output is never the same twice

Snapshot diffs are useless against a model, because the same prompt legitimately produces different sentences. The fix is to assert on structure and behaviour rather than exact text: an answer appears, it references the uploaded document, the citation opens, the input stays enabled for a follow-up. A user could verify each by looking, and each holds across paraphrases.

TesterArmy tests are plain-English steps with one intent each, actions split from assertions, following our guide to writing test steps. The guide's own example of the right shape rewrites "Wait for AI to finish and check the output" into "Wait for the generated summary to finish and verify it contains three bullet points." Here is a full journey for a retrieval-backed assistant, written the same way:

Seven steps, one flow, and none depends on the model's wording. The revenue step accepts any figure with a currency symbol because the point is that a number was retrieved and presented; the precise value belongs in an eval. The citation step catches the dead-link failure above. The last step catches the lost-state failure, because an input that is disabled after the first answer is a broken product regardless of how good the answer was.

Tool calls, refusals, and latency

For agentic flows, assert on the end state a user can see. If a support assistant is meant to process a refund through your support tool, the journey is: ask for a refund on a specific order, verify the assistant confirms it, then open the Orders page and verify that order shows as refunded. The last step matters most, because it fails when the tool call silently errored and the assistant confirmed anyway. Refusals get the same treatment. Ask a question the uploaded document cannot answer, and verify the assistant says the document does not contain that information rather than producing a figure. A model that invents a number here passes a fluency eval and fails your user.

Latency and streaming are journey properties, so they belong in the journey test. TesterArmy treats a stream as one step: assertions run on the final concatenated output, and latency assertions split into a first-token budget and a completion budget, so a stalled stream or a slow first token is caught without depending on the model's words. In plain English that is a step like "Verify the assistant starts responding within four seconds and finishes the answer", with the budget set to what your product promises rather than the model's best day.

Calibrating strictness

The craft is deciding where to be strict. Be strict about meaning where the value matters: the order is marked refunded, the source named is the one uploaded. Be tolerant about presentation where phrasing changes often: "a confirmation message appears" rather than the quoted sentence. Prefer positive observations over absences, because "no error appears" passes for many wrong reasons, including a page that never loaded. And avoid vague verdicts such as "verify the answer is helpful", which no reader, human or agent, can evaluate consistently. Our guide to writing test cases in plain English has the longer version of these rules, and they apply unchanged to AI products.

Where the agent sits next to your eval stack

It does not replace it. Evals catch model regressions across hundreds of prompts at once; a journey test runs one flow at a time against the deployed product and returns a verdict with evidence. The eval owns answer quality on a dataset, the journey test owns whether the product around the model still works for a user, and most AI teams need both.

Because the agent looks at the rendered page, the failures above are visible to it the way they are visible to a user. That is also why the model driving the agent matters: reading a screen and clicking the right place on it are separate capabilities, and the numbers are in our benchmark of ten vision models on clicking what they see. On every pull request the exploration agent reads the diff, which includes a changed system prompt, and writes a plan aimed at that change; how that plan compares with a generic generated test case is in can AI generate good test cases. If the coding agent that edited the prompt is Claude Code or Codex, it can queue the journey itself and read the verdict back, as shown in using Claude Code and Codex to create and run e2e tests. Teams that first handed their coding agent a raw browser will recognise the distinction in Playwright MCP vs an AI testing agent.

The evidence makes the verdict actionable: a recording of the journey, a step trace with per-step results, screenshots at each key moment, and a check plus comment on the pull request that introduced the regression. Sarup Banskota, Co-Founder & CEO of CodeCrafters, put it this way: "Finally something that allows me to confidently ship my heavily prompt engineered prototypes to live customers."

Two boundaries. The agent does not measure answer accuracy at scale; if your average grade dropped across five hundred prompts, only an eval will tell you. And it does not judge tone or taste, so whether the assistant sounds right for your brand stays a human review, ideally of the recordings the agent produces.

FAQ

Can you test an LLM chatbot end to end without matching exact output? Yes. Write assertions on structure and behaviour: an answer finished streaming, it names the expected source, the citation opens, the follow-up input is enabled, the Orders page shows the refund. Those conditions hold across paraphrases, and a plain-English step can express each of them without quoting the model.

Do end-to-end tests for AI apps replace evals? No. Evals score the model against a dataset and catch regressions in answer quality. Journey tests run the deployed product as a user and catch the failures around the model: streaming, tool calls, retrieval links, session state, latency. Teams shipping AI products need both.

How do you test streaming responses? Treat the stream as one step. Assert on the final concatenated output for content, and set a first-token budget and a completion budget for latency. That catches a stalled stream, a hang in the middle, or a slow first token without depending on the wording of the answer.