Software testing in both production and non-production environments have merits and demerits. There is no right answer to the question of which is a better method. Businesses can make choices for either testing processes or a combination of both. They go by the business requirements, IT risk and security assessments.
Production testing is performing tests in a live environment after production deployment.
We list some pros of testing in production environments:
Real World Accuracy
Testing production systems acquaints testers with reality. Testing in non-production environments like development, quality assurance, and staging systems has value but it does not mirror the real world. This is because they are running on a different code, server operating system, and patch levels. Often testing in the two environments will yield starkly different results, especially noticeable for vulnerable and penetration testing, as configurations vary at the application and server levels.
Smoke and sanity tests are mostly performed in production environments. Smoke testing checks that the product’s main functionalities are working correctly. Sanity testing verifies if bugs have been fixed and new functionality is performing as expected.
Performing usability testing in a production environment can give businesses an understanding of the user experience. A/B testers offer two versions of a product to users and find statistics to determine which version performs better. A/B testing method of testing is ideal to test for new features, whose market value is unknown.
No matter how much testing is done in development, bugs will reach production. If the DevOps team is prepared to handle defects at the production stage, the recovery time will be faster. Hence, testing in production helps in managing disastrous situations, by taking testing a step ahead of mere prevention techniques, to monitoring methods that detect and fix bugs continuously. The application gains resilience and can recover from unexpected chaos without loss of functionality or data.
Complete test coverage
As conditions of a production environment cannot be fully replicated, complete test coverage is not possible before product deployment. Due to data and traffic limitations, many edge case tests in staging environments will be missed out. Testing in production will give full test coverage via continuous monitoring. With production data, every possible scenario can be detected. Also, developers often make changes in applications and deploy to production, skipping the Quality Assurance verification. These changes create bugs in the application. Testing in production with software testing tools removes the risks caused by these changes.
We now list some cons of testing in production environments:
Irresponsibility in testing
Testing in production helps the overall testing process but it is not meant to lessen the tester’s responsibility for testing in other environments. As production testers share the responsibility of testing, QA testers may develop an unmotivated approach and not utilize the saved time towards other types of testing. Testing in production will only be effective if team members collaborate fully.
If a load test is run on the production environment during business hours, it increases the chances for poor user experiences. Experts recommend running tests in production environments during times that do not impact a lot of users i.e. night or maintenance hours. Testers get a smaller time slot to perform the tests. As user experience is tied with profit, a tester not adhering to the recommended timings, can lower the business revenue.
Chaos Monkey is a code with which chaos is created on purpose in a production environment. It is a technique to verify if production is working as expected. However, a code that injects failures deliberately can cause problems in an environment like damaging a web server. Chaos monkey should be used carefully when developing a resilient application to ensure application tiers are not killed.
If correctly done, testing in production is a wonderful opportunity, bringing the product team closer to the end-users. The scope of testing in production has not been fully tapped into, yet. Exploring the potential of testing in production is a definite feature of the future.