How to Perform Post-Release Testing Effectively and Minimize Impact of the Release to Live Clients

When I started my career as a QA, I was working with a company that offered its products as SaaS. Production releases were critical and there was a possibility of affecting the functionality for the live clients.

As our client base grew, to manage the risk and minimize the impact of the release to live clients, QA team adopted the post-release testing practice.

This was all new to me and I had so many questions and doubts in my mind: 

• What is post-release testing?
• I tested everything properly, why do we need to do post-release testing?
• Do I test everything all over again? What do I exactly do in post-release verification?
• What happens if I find an issue? Etc.

I am happy to admit that I found all my answers within my first few production releases.

Here I am sharing that knowledge with you all.  I chose to write the article in a question and answer format to show you the way I discovered the answers.

What is Post Production Release Verification?

By definition, Post means After, Production Release refers to deployment to LIVE/production environments and Verification includes making sure the features released meet the requirements.

Recommended read => How to Effectively Prepare “Test Environment” before starting to test

The objective is to verify the release on production/LIVE environments.

But the questions then arise:

• Why do we need to do post production release testing when I tested everything on QA environment?
• Why do we anticipate issues to occur on production although we tested the release thoroughly on test environment?

There are many reasons why we would have issues on production even though we might have followed complete Quality assurance process (i.e. test planning, test plan review, test cycle, regression tests etc.)

Reasons why we would have issues on production:

1) Data Issue – The data available on production and test environments may vary. This can cause some corner-case issues to be missed on test environments.

2) Deployment issue – If your company has a manual build deployment process, your release may be more prone to deployment issues. Some common scenarios can be, missing configuration or site settings, missing DB scripts, order of deployment not followed (code first, then DB etc.), dependencies installed incorrectly, etc.

Also read => What QA Tester Should Know About Deployment process

3) Impact areas not identified – There can be some scenarios for which the impacted areas may not have been identified correctly and completely by the team.

For example, consider a SaaS environment.

If the team did not identify the impact of a re-factored DB table on a client using older table schema (e.g. data loss, need for data migration before release, etc.), etc. This issue is less likely to happen for well-planned projects with precise requirements. But, the possibility still exists.

4) Unknown Impact areas – This can occur if the scope and impacted areas of the release are not known. For example, in a company with several software products  sharing common DB and architecture, even a small change can break the functionality of many products.

What tasks and activities are included in post-release verification phase?

Post production release tasks and activities generally include:

• Post production release verification
• Report verification results
• Reporting any issues found on production
• Post release verification data clean up
• Post release monitoring (if applicable)

Do I need to test everything all over again?

Not necessarily. This depends on the build to be released and the impact analysis.

Detailed testing should be done during regular QA cycle. Post release verification should be done by following a Post production release verification test plan that should be a derivative of the full Test Plan for that release.

How do I formulate post production release verification strategy?

Post production release verification planning needs to be done in a similar way as your regular test planning.

The strategy should be on the same lines as the test flow followed during the QA cycle. It is important to include most important and critical steps that allow maximum functionality coverage.

A good post production release strategy should:

• Include steps to test new features as well as major existing features
• Verify major impact areas
• Allow maximum functionality coverage
• Optional: Include any critical bugs that were found in test environment
• Optional: Include priority of the test cases

Who creates the post production release test plan?

This will vary across companies and will depend on the organization structure.

Let’s take an example of the following QA team organization.

In this scenario, QA working on the specific project will formulate the initial post production release test plan.

Who approves the post production release test plan?

This will vary across companies and will depend on the organization structure.

Again considering the same organization structure as shown in the previous question, the post production release test plan should be reviewed and approved by the Test Lead or QA Manager.

When do I create the post production release verification plan?

The post production release test plan can be created anytime during the software development cycle after the requirements, development scope and impact areas are identified and locked. It is usually easier for the QA to create the post production release test plan midway into the sprint. That ensures that there is enough time for review and approval.

It is a good practice to include this test plan along with any formal QA sign off documents before the project enters the deployment and release phase.

I completed the post production release verification. What’s next?

After the post release verification is complete, the next steps would be

1) Communication of verification results – The verification results should be communicated to the stakeholders including any issues that may have been found on production.

2) Reporting any issues found on production in the Defect Management tool – To facilitate root cause analysis and traceability.

3) Post release verification data clean up – The data cleanup needs to be done after verification is complete.

For example, consider a release for an eCommerce application and say you created a test order on production. This test order needs to be canceled after the verification is complete.

4) Post production release monitoring (if applicable) – Some releases require monitoring on production.

For example, if the team made improvements to improve the page load times in the Application, this would need to be monitored over some period of time to ensure that the improvement was indeed seen after release. The responsible person(s) for monitoring should be clearly identified and communicated to.

What happens if I find an issue?

Any issues should be reported in the Defect management tool and communicated to the stakeholders. If any critical issues are found on production, communication of results should occur immediately as a decision would need to be made if the build needs to be rolled back to investigate the issue further.

It is important that all the issues found are reported in the Defect Tracking Tool. It is recommended that these be raised as separate issue type (e.g. Post Production Bug) to show separation from regular QA cycle bugs. These issues can be filtered out easily if required for the purpose of root cause analysis.

What else do I need to know about post production release verification process?

Besides the actual post production release verification process, plan and strategy, below are some pointers:

• It is important to set clear expectations regarding scope and purpose of post release verification. Stakeholders (internal and external) should be made aware of the following
  • Team cannot test everything on production
  • Team cannot squeeze days worth of testing into few hours set aside for post release verification

Therefore, the testing on production would be essentially based on approved post production release test plan.

Limitations:

Due care should be taken while deciding the extent of post-production release testing. There are limitations to what and how much we can actually test on production. Production environment has live client data and needs to be handled very carefully. Additional planning should be done for changes that involve data migration, updating, deletion etc.

Example #1): For an eSurvey company, if testing involves answering and submitting the survey, QA would need to send a request to delete the test survey after verification so as to not impact the client survey collection data and their statistics.

Example #2): For an e-commerce company, let’s assume a pricing update SQL job runs at midnight every day and uploads the finalized price to the website. We cannot run this SQL on-demand, multiple times for the purpose of post-release verification as this may cause unfinalized data to be pushed to production.

Moreover, it can increase the chances of DB deadlocks and high consumption of CPU and memory resources during peak business hours which can affect the client application performance.

• The effort required for post-release testing and all related activities should be inbuilt and included in the Project Plan. Depending on the business rules and project specifics, this can be considered as project overhead or included in QA cycle or included as part of the release management plan.
• For the issues that are reported during post-release verification, root cause analysis should be conducted to find out the reason why the issue was not caught early on and what can be done better next time to avoid facing the issue. The root cause analysis can help the team to learn from these past issues and fill in any gaps in the implementation. Based on the organization structure, the Test Lead or QA Manager can complete the Root cause analysis with input from the project team. Some common root causes can be a Coding issue, requirements issue, design issue, data issue, 3rd party limitations, missing test scenario etc. Corresponding Corrective and Preventive Actions can be created and tracked.
• Server Logs can also be used to monitor the build after release. Server log may contain events or issues that may not be visible to the customer but will cause issues in the backend. This monitoring can be assigned as an action item to Dev lead and DevOps team.

An Example:

Project Overview:

Following changes need to be made to a social media application, specifically to the sign up process

1. Last name field validation needs to be removed. It was previously implemented as ‘Last name should have minimum 4 characters’ (Improvement for existing field)
2. Implement toggle button next to email address so that user can set the privacy settings for email address to show on their profile (new feature request)
3. User should be able to choose their avatar (new feature request)
4. Reduce API calls during Sign up process to improve application performance (Improvement)

Post production release verification Plan:

Conclusion

With most of the software companies now adopting the Agile methodology, the number of production releases has increased.

For example, while using Waterfall model, a team may have a production release every 1.5 months, however with the Agile process, the same team may now have a production release every 2-3 weeks.

With every production release, we have a possibility of knowingly or unknowingly impacting the functionality of the live clients. The adoption of post production release verification immediately after release can provide additional confidence on the release at the same time providing the safety net of rolling back the release before our live clients come across some issues.

For high impact/risk projects, post-production release verification plan can be structured based on the priority of the test scenario. Critical priority test can be executed first and communication sent to stakeholders about results and any issues. If no critical issues are found, then the post production release verification can continue, otherwise, the decision for roll back needs to be made to minimize application downtime and impact to live clients.

Additionally, post-production release testing can be automated and the test scripts can be run on demand after every release as a Regression test. Again, due care should be taken while running the automated test scripts on production as it may affect live client data and functionality.

Post production release verification is the last line of defense for any software company. If we do not catch the issues, our customers will and this can be devastating to the reputation of any Software company.

In order to maintain the reliability of the product, it is essential that we verify the changes deployed to production immediately after deployment.

About the author: This helpful article is written by Neha B. She is currently working as a Quality Assurance Manager and specialize in leading and managing In-house and Offshore QA teams.

Share your post-production release testing strategy / tips / experience with our readers.