Smoke Testing vs Regression Testing: What Runs When
Smoke tests ask if the build is alive; regression tests ask what broke. The real differences, when each runs, and how to wire both into PRs and deploys.
Smoke and regression testing get confused because both run the same kind of checks against the same app; the difference is the question being asked and the moment it is asked. Getting the split right matters practically: teams that blur it either gate every deploy on an hour of tests (and start skipping them) or ship on a five-minute smoke pass and call it regression coverage (and meet their bugs in production).
The two questions
Smoke testing asks: is the build fundamentally alive? The name comes from hardware, where you power a board and watch for smoke. A software smoke suite is small and brutal: the app loads, login works, the one flow the business exists for completes. It runs in minutes, immediately after every build or deploy, and a failure means stop, since nothing else you test matters if the app cannot start.
Regression testing asks: did this change break anything that used to work? It is broad by definition, covering the flows you have promised yourself will keep working: checkout and billing, settings, permissions, the unglamorous paths users rely on. It runs before releases, on a schedule, or (ideally) on every pull request, and a failure means a specific promise got broken by a specific change.
The differences that actually matter
| Smoke | Regression | |
|---|---|---|
| Question | Is it alive? | Did we break anything? |
| Scope | 3-10 critical checks | Every flow worth protecting |
| Runtime | Minutes | As long as coverage demands |
| Trigger | Every build/deploy | Every PR, schedule, or pre-release |
| On failure | Halt everything | Fix before merge/release |
| Depth | Happy path only | Variants and edge paths included |
The most useful mental model: smoke is a subset of regression promoted to run first and always. Every smoke check is also a regression check; the reverse is deliberately false.
Where each runs in a modern pipeline
The classic pattern is layered. A pull request triggers the regression flows relevant to the change; the deploy to staging triggers smoke immediately; production deploys get smoke plus a scheduled regression pass against live. What broke this pattern historically was cost: when tests were hand-written code, a "full regression suite" was expensive to build and slow to maintain, so teams quietly shrank regression to a smoke suite and hoped. The economics that caused that shrinkage are the subject of the hidden tax of test maintenance.
Agent-based testing changes the cost side. With TesterArmy, both layers come from the same plain-English flow definitions, organized into groups: a "smoke" group with the three or four vital flows wired to run on every deploy, and a "regression" group with everything else running on every pull request and nightly. The agent performs each flow in cloud browsers or mobile simulators and posts results to the PR with recordings and step traces, so both the fast answer and the deep answer arrive without anyone maintaining two test codebases. PR-level regression, the layer teams most often skip, is covered in depth in Test Every Pull Request, and the always-on production layer in Testing in Production.
Sizing each layer honestly
For smoke: if a check failing would not halt a deploy, it does not belong in smoke. Most products need fewer than ten checks, and the discipline is keeping the list short as everyone lobbies to add theirs.
For regression: inventory the flows whose breakage would generate a support ticket within a day. That list, which is usually 20 to 60 flows for a product team and far fewer than "everything", is your regression scope. Flows that would take weeks to encode as test code take an afternoon to describe in plain English, which is what makes the honest scope reachable rather than aspirational.
FAQ
Is smoke testing the same as sanity testing? They are cousins: smoke verifies a build is broadly alive, sanity is a narrow post-fix check that a specific change behaves. In practice most teams fold sanity checks into their PR-level regression runs.
Should smoke tests run in production? Yes, and continuously rather than only at deploy time, since production breaks between deploys too (third-party changes, expiring certificates, data issues). That is production monitoring, and it can reuse the same smoke flows.
How many regression tests do we need? As many as there are flows whose breakage you would fix urgently. Count those; that is the number. Adding more than that adds runtime without adding protection.
