Testing in Production: How to Validate Software Safely

Testing in production means running validation checks against a live system with real users and real data, rather than only in pre-release environments. The safe way to do it is to limit blast radius, control traffic, and monitor closely, so you can detect and fix problems before they affect many users. This article explains the main techniques, their risks, and how to apply them responsibly.

Why Test in Production at All?

Traditional staging environments try to mirror production, but they never fully match it. Real production has unpredictable user behavior, different data distributions, network latency, and third-party integrations that are hard to simulate. Testing in production gives you the most accurate feedback on how a change behaves under actual conditions. It also lets you verify performance, security, and user experience with real traffic.

However, the phrase “testing in production” often scares people because it sounds like you are experimenting on real users. The key is to do it safely, using techniques that minimize risk and provide a quick way to revert.

Core Techniques for Safe Validation

Several established patterns allow you to test in production without causing widespread harm:

Canary releases: Route a small percentage of traffic to the new version, monitor, then gradually increase. If something goes wrong, you roll back quickly.
Feature flags: Deploy code that is hidden behind a flag. You can turn it on for a small group, then for everyone, without redeploying.
Shadow traffic: Send a copy of real requests to the new version but do not return the results to users. Compare the outputs to the old version.
A/B testing: Split users into groups and show different versions. This is more for product decisions but also validates behavior.
Blue/green deployment: Run two identical environments, switch traffic from blue to green, and if problems appear, switch back.

Each method has trade-offs. Canary releases are the most common for backend services. Feature flags are great for frontend changes. Shadow traffic is useful for data-heavy systems but can double your load.

How to Choose the Right Approach

Your choice depends on what you are changing and what you can measure.

For a small bug fix in a web API: A canary release with 1-5% traffic is enough.
For a new UI component: Use a feature flag and enable it for internal users first, then a small percentage of external users.
For a database migration: Shadow traffic or a blue/green approach with a full rollback plan is safer.
For a machine learning model: A/B testing with clear metrics (e.g., click-through rate) is typical.

Always start with the smallest possible exposure. You can always increase traffic later, but you cannot undo a bad experience for a user who already saw it.

Common Problems and How to Avoid Them

Even with the best intentions, testing in production can fail. Here are common pitfalls:

Not having good monitoring: If you cannot see errors in real time, you might not notice a problem until it affects many users. Set up dashboards for error rates, latency, and business metrics.
Ignoring data consistency: If your new version writes data in a different format, you might corrupt production data. Use shadow traffic or a separate database for testing.
Rolling back too slowly: A rollback that takes 30 minutes can feel like an eternity. Automate rollback as much as possible.
Testing everything at once: If you change multiple things simultaneously, you won’t know what caused a problem. Isolate variables.
Not considering user impact: Some tests, like load testing, can degrade performance for real users. Run those in off-peak hours or on a separate environment.

The most important rule is: always have a rollback plan. Before you start, know exactly how you will revert the change, and test that rollback in a safe environment.

Comparison: Testing in Production vs. Staging

Staging environments are not useless; they are the first line of defense. They catch obvious bugs and integration issues. But they have limitations:

Data: Staging often uses synthetic or anonymized data, which may not reflect real edge cases.
Scale: Staging rarely has the same hardware or traffic volume as production, so performance issues may not appear.
Dependencies: Third-party services may not be available or may behave differently in staging.

Production testing complements staging, not replaces it. Use staging for basic checks, then use canaries or feature flags for final validation.

Costs and Risks

Testing in production has real costs. You need extra infrastructure (e.g., a second environment for blue/green), more monitoring tools, and engineering time to design and maintain the testing pipeline. There is also the risk of user-facing errors, which can damage trust and revenue.

To mitigate these, start small and use automated checks. For example, a canary release might automatically roll back if the error rate exceeds 0.1% for five minutes. This reduces the need for human intervention and limits the blast radius.

Another cost is the cognitive load on developers. They need to understand the testing framework and be ready to respond to alerts. This is why many teams adopt a “shift-left” approach: test as much as possible in earlier stages, and use production only for the final verification.

Alternatives and Complementary Practices

If you cannot test in production safely, consider these alternatives:

Production-like environments: Build a staging environment that closely mimics production, including realistic data and traffic simulation.
Chaos engineering: Intentionally inject failures in a controlled way to test resilience. This can be done in production but with strict guardrails.
Post-deployment monitoring: Even without canaries, you can monitor closely after a full rollout and be ready to roll back.

There is no one-size-fits-all solution. A mature team might combine several techniques: use feature flags for UI changes, canaries for backend services, and chaos engineering for infrastructure resilience.

Frequently Asked Questions

Is testing in production illegal or unethical?

No, it is a common practice in software engineering. The ethical concern is when you test without user consent or without safety measures. As long as you minimize impact, have a rollback plan, and comply with privacy laws (e.g., not exposing personal data), it is acceptable.

What is the difference between canary and A/B testing?

Canary testing is about deploying a new version to a small subset of users to check for technical issues. A/B testing is about comparing two versions to see which performs better on business metrics. Canaries are usually about stability; A/B tests are about optimization.

How much traffic should I send to a canary?

It depends on your traffic volume. For a high-traffic service, 1% might be enough to detect errors. For a low-traffic service, you might need 10% or more to get statistically meaningful results. Start with the smallest percentage that gives you enough data.

What if I don’t have a rollback plan?

Then do not test in production. A rollback plan is non-negotiable. At minimum, you should be able to revert to the previous version within minutes, either by redeploying the old artifact or using a feature flag to disable the change.

Can I test in production on a mobile app?

Yes, but it is trickier because you cannot control which version users have. Use feature flags that are fetched from a server, so you can turn features on or off remotely without requiring an app update.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →