Testing in Production Without Breaking Production
Production is the only environment that tells the whole truth. Which flows to test against live, which never to, and how continuous journey checks work.
"We don't test in production" was always half a joke, because everyone does; the only question is whether it happens deliberately, with safe flows and alerts, or accidentally, with customers as the test runners. The serious case for deliberate production testing is simple: staging can only tell you about the code, while production is where the code meets real DNS, real third parties, real data volumes, and real configuration. Some failures exist only there, and no amount of pre-merge rigor sees them.
What only production can tell you
The failure classes that live exclusively in production read like a list of things that were "green in staging": a payment provider's silent API change, an expired certificate or key, a CDN misconfiguration serving stale bundles, a feature flag flipped in the live dashboard but never in staging, third-party scripts degrading the checkout, and behavior that only emerges at production data scale. None of these are caused by a code change, so PR-level testing structurally cannot catch them (that layer has its own job, covered in Test Every Pull Request); they appear between deploys, on their own schedule, which is why the answer is continuous rather than event-driven.
The safety rules, because production is live
Testing against live systems is a discipline with hard edges:
- Read-mostly flows. Verify browsing, search, sign-in, dashboards; do not exercise destructive paths (deletions, refunds, mass emails) against production. Save those for staging's nightly regression.
- Dedicated test accounts, clearly marked. Excluded from analytics, business metrics, and marketing automations, so your tests never look like customers or email one.
- Purchases stop before money moves, unless you have a test-mode path with your payment provider that is safe end to end. "Add to cart and reach the payment form" verifies most of the funnel with none of the risk.
- Respect your own rate limits and error budgets. A production check every few hours is monitoring; one every ten seconds is a self-inflicted load test.
- Alerts must reach humans fast, or the whole exercise is a dashboard nobody reads.
Synthetic journeys: the practical form of production testing
The workable pattern is synthetic monitoring done at journey level: an agent performs real user flows against production on a schedule and alerts when one breaks. This is distinct from uptime pings (which confirm the server answers, and happily return 200 while checkout is broken) and from error tracking (which catches exceptions, while the worst production bugs return clean responses that simply do not do the job; our web solutions page calls this the journey-level gap).
With TesterArmy, the setup reuses what you already have: the same plain-English flows that run on pull requests get a schedule and a production target. The agent signs in with test accounts, walks the critical journeys in a real browser around the clock, and when something breaks your team gets a Slack message with the flow, the failed step, and a recording. Two of our customers describe the operating feeling on our site: Standout's CTO monitors a logged-in dashboard continuously after automatic auth with test accounts had been the blocker that killed earlier attempts, and Lightsprint's co-founder stopped manually checking production after each deploy, waiting for the Slack message instead.
What belongs in the production set
Keep it curated: the sign-in, the money path up to its safe boundary, the core loop your product exists for, and the surfaces where third parties can hurt you (embedded scripts, OAuth providers, email delivery). For most products that is five to ten flows. Everything else runs against staging, where destructive paths are safe and coverage can be broad. The scheduling split (smoke cadence against production, full depth nightly against staging) follows the smoke versus regression logic.
FAQ
Is testing in production bad practice? Untargeted, destructive, or anonymous-to-analytics testing in production is bad practice. Curated read-mostly journeys with marked accounts and alerting is standard practice at mature teams, usually under the name synthetic monitoring.
How is this different from uptime monitoring? Uptime checks ask whether the server responds; journey checks ask whether a user can accomplish the thing. Products are routinely "up" and broken at the same time, and only the second kind of check notices.
How often should production flows run? Match the cost of downtime: every few hours suits most products, hourly for revenue-critical funnels. The constraint is your own systems' comfort rather than the testing tool.
