Self-Healing Tests: What Is Real and What Is Marketing

"Self-healing tests" now appears on the product page of nearly every test automation tool, and it describes at least three different mechanisms with very different consequences for the team that buys them. We build TesterArmy, an agent that tests apps like real users, and we get lumped into the phrase often enough to want to sort it out. The short version is that some products heal locators, some propose repairs for a human to accept, and some, including ours, never store a locator that could break in the first place. Each of those is a real capability. Each also gives something up, and the marketing tends to leave that half out.

Why the phrase exists at all

A conventional end-to-end test finds elements by structure: a CSS path, a test ID, an XPath, a role and a label. The product is maintained in the language of meaning, so structure moves constantly and the test breaks even though nothing a user would notice has changed. We covered the mechanics in why tests break on every UI change and the operational cost in cutting Playwright test maintenance. Every variant of self-healing answers that single problem: the encoded structure drifted, and something has to re-bind the test to the page. Vendors differ in who does the re-binding, when, and whether anything is stored afterward that can drift again.

Level 1: locator fallback

The oldest mechanism records several ways to find each element when the test is authored or first run: the ID, the nearest label, the visible text, a relative XPath, sometimes a position. When the primary locator fails at run time, the tool tries the alternatives in order, and if one matches, the step proceeds and the run is marked as healed. It removes a category of red builds that were never about the product.

Its limitation is what happens next. The suite still exists, the broken locator is still in it, and the fallback is a guess: the text "Continue" might now match a different button than the one the author meant. Most tools report healed steps, but a team that does not read those reports accumulates drift silently until a fallback lands on the wrong element and a test passes for the wrong reason. Level 1 makes each breakage cheaper without reducing the number of encoded bets, and it adds a review job, the healing report, on top of the old one.

Level 2: AI re-identification plus a proposed patch

The newer variant replaces the fallback list with a model. When a locator fails, the tool takes the current page, the failing step, and sometimes a screenshot from the last passing run, and asks a model to identify the element the step most plausibly meant. It then proposes a change to the test code or specification, and a human approves it as a pull request or a diff.

This improves on level 1 in two ways: the re-identification is semantic rather than a fixed list, and the approval step means the suite does not drift without anyone knowing. The costs are also real. Someone still reviews and merges patches, which is the maintenance job in a lighter form, and the suite still lives in your repository as code that encodes structure. The repair loop is shorter; it has not gone away. If your team values having tests as code you can read, diff, and run offline, level 2 is likely the right trade. Tools that generate a suite and then heal it belong here too, and how much authoring they can genuinely take over is the subject of can AI generate good test cases.

Level 3: nothing stored that could break

The third mechanism is the one TesterArmy uses, and strictly speaking it is not healing, because there is nothing to heal. A test is a list of plain-English steps such as "Open the Pricing page from the header" and "Verify the cart shows the Pro plan with the correct monthly price", each with one intent; our guide to writing test steps is the reference. No selector, component name, or coordinate is stored with the step. On every run the agent looks at the rendered page, decides what to click from the visible UI, performs the action, and checks the result, relying on vision rather than DOM matching. If the button moved, was restyled, or the DOM around it was rebuilt, there is no encoded expectation to violate, so the run proceeds. If the button is gone or the flow no longer works, the step fails, which is the failure you wanted.

Memory adds a second layer without reintroducing brittleness. The agent keeps project memory across runs, notes like where login lives or which account to use for billing flows, and every run feeds it, so the agent gets sharper about a particular app over time. None of that memory is a locator.

The word "heal" describes this badly. A level 1 or level 2 tool detects that a stored expectation broke and repairs it. A level 3 agent never formed the expectation, so there is no repair event, no healing report, and no patch to approve. That is why "self-healing" applied to an agent is a marketing convenience rather than a description. The phrase travels: the Playwright MCP README, as of September 2026, lists self-healing tests among the loops MCP suits, meaning a coding agent re-reading page structure on each pass. What separates raw browser control from a hosted testing agent is covered in Playwright MCP vs an AI testing agent.

What level 3 gives up

Anyone selling level 3 should say this out loud. Determinism per step is looser. A scripted click on #checkout either fires or fails identically every time; an agent reading the page can take a slightly different path through the same step on two runs. TesterArmy constrains this by executing one step at a time with one intent each and returning a step trace, per-step screenshots, and a video, so the path taken is inspectable rather than trusted. When the agent cannot finish a step within its action or time budget, the run ends BLOCKED with an automation reason instead of a guessed verdict, and blocked runs are excluded from pass rates.

Offline diffing is gone, because there is no code suite in your repository to diff, review in a pull request, or run locally without the service. Tests are bounded: a test cannot have more than 30 steps and shares one time budget across them, so long flows are split into groups rather than written as one script. Network mocking, seeded fixtures, and contract-level API assertions stay framework territory, as we said in autonomous QA: can an agent replace your test suite. The same accounting applies on mobile, where testID matchers play the role of selectors; see why teams are replacing Detox. Where a locator-free layer sits relative to unit and integration tests is taken up in is the test pyramid still relevant.

Questions to ask any vendor using the term

The term will not tell you which level you are buying, so ask about the mechanism. What is stored per step, and what is compared against it at run time? When a heal happens, who is told, and what do they have to do about it? Can a heal land on the wrong element and still report a pass, and how would you find out? Is there a suite in your repository afterward, and who owns it? What does a run return when the tool cannot decide, and is that state separate from a product failure? And what maintenance surface remains once the healing works as advertised?

A level 1 vendor will describe fallback lists and a report, a level 2 vendor patches and a review queue, and a level 3 vendor should describe a step trace, a blocked state, and the limits above. Any of those can be the right purchase. The wrong one is made because the same two words were on three very different boxes.

FAQ

Are self-healing tests worth it? Yes, if you know which mechanism you are buying. Locator fallback and AI patching both cut the repair bill for a suite you keep owning, and they suit teams that want tests as code in the repository. An agent with no stored locators removes the repair bill entirely and trades away per-step determinism and offline diffing. Match the mechanism to which cost you are trying to stop paying.

Does TesterArmy have self-healing tests? TesterArmy stores plain-English steps and no locators, so there is nothing to heal and no healing event. The agent reads the rendered page on every run. A step fails when the meaning of the page has changed, for example the button is gone or the flow no longer completes, and a run is marked BLOCKED rather than FAILED when the environment or the agent's own limits stopped it.

Can self-healing tests hide real bugs? At level 1, yes: a fallback locator can match the wrong element and the step passes for the wrong reason, which is why healing reports need a reader. Level 2 reduces this with human approval. Level 3 carries a different risk, an agent finding a valid path you did not intend, which is why every TesterArmy run returns the step trace, screenshots, and video so the path can be checked.