When I started my career as a QA, I was working with a company that offered its products as SaaS. Production releases were critical and there was a possibility of affecting the functionality for the live clients.
As our client base grew, to manage the risk and minimize the impact of the release to live clients, QA team adopted the post-release testing practice.
This was all new to me and I had so many questions and doubts in my mind:
I am happy to admit that I found all my answers within my first few production releases.
Here I am sharing that knowledge with you all. I chose to write the article in a question and answer format to show you the way I discovered the answers.
What You Will Learn:
By definition, Post means After, Production Release refers to deployment to LIVE/production environments and Verification includes making sure the features released meet the requirements.
Recommended read => How to Effectively Prepare “Test Environment” before starting to test
The objective is to verify the release on production/LIVE environments.
But the questions then arise:
There are many reasons why we would have issues on production even though we might have followed complete Quality assurance process (i.e. test planning, test plan review, test cycle, regression tests etc.)
Reasons why we would have issues on production:
1) Data Issue – The data available on production and test environments may vary. This can cause some corner-case issues to be missed on test environments.
2) Deployment issue – If your company has a manual build deployment process, your release may be more prone to deployment issues. Some common scenarios can be, missing configuration or site settings, missing DB scripts, order of deployment not followed (code first, then DB etc.), dependencies installed incorrectly, etc.
Also read => What QA Tester Should Know About Deployment process
3) Impact areas not identified – There can be some scenarios for which the impacted areas may not have been identified correctly and completely by the team.
For example, consider a SaaS environment.
If the team did not identify the impact of a re-factored DB table on a client using older table schema (e.g. data loss, need for data migration before release, etc.), etc. This issue is less likely to happen for well-planned projects with precise requirements. But, the possibility still exists.
4) Unknown Impact areas – This can occur if the scope and impacted areas of the release are not known. For example, in a company with several software products sharing common DB and architecture, even a small change can break the functionality of many products.
Post production release tasks and activities generally include:
Not necessarily. This depends on the build to be released and the impact analysis.
Detailed testing should be done during regular QA cycle. Post release verification should be done by following a Post production release verification test plan that should be a derivative of the full Test Plan for that release.
Post production release verification planning needs to be done in a similar way as your regular test planning.
The strategy should be on the same lines as the test flow followed during the QA cycle. It is important to include most important and critical steps that allow maximum functionality coverage.
A good post production release strategy should:
This will vary across companies and will depend on the organization structure.
Let’s take an example of the following QA team organization.
In this scenario, QA working on the specific project will formulate the initial post production release test plan.
This will vary across companies and will depend on the organization structure.
Again considering the same organization structure as shown in the previous question, the post production release test plan should be reviewed and approved by the Test Lead or QA Manager.
The post production release test plan can be created anytime during the software development cycle after the requirements, development scope and impact areas are identified and locked. It is usually easier for the QA to create the post production release test plan midway into the sprint. That ensures that there is enough time for review and approval.
It is a good practice to include this test plan along with any formal QA sign off documents before the project enters the deployment and release phase.
After the post release verification is complete, the next steps would be
1) Communication of verification results – The verification results should be communicated to the stakeholders including any issues that may have been found on production.
3) Post release verification data clean up – The data cleanup needs to be done after verification is complete.
For example, consider a release for an eCommerce application and say you created a test order on production. This test order needs to be canceled after the verification is complete.
4) Post production release monitoring (if applicable) – Some releases require monitoring on production.
For example, if the team made improvements to improve the page load times in the Application, this would need to be monitored over some period of time to ensure that the improvement was indeed seen after release. The responsible person(s) for monitoring should be clearly identified and communicated to.
Any issues should be reported in the Defect management tool and communicated to the stakeholders. If any critical issues are found on production, communication of results should occur immediately as a decision would need to be made if the build needs to be rolled back to investigate the issue further.
It is important that all the issues found are reported in the Defect Tracking Tool. It is recommended that these be raised as separate issue type (e.g. Post Production Bug) to show separation from regular QA cycle bugs. These issues can be filtered out easily if required for the purpose of root cause analysis.
Besides the actual post production release verification process, plan and strategy, below are some pointers:
Therefore, the testing on production would be essentially based on approved post production release test plan.
Due care should be taken while deciding the extent of post-production release testing. There are limitations to what and how much we can actually test on production. Production environment has live client data and needs to be handled very carefully. Additional planning should be done for changes that involve data migration, updating, deletion etc.
Example #1): For an eSurvey company, if testing involves answering and submitting the survey, QA would need to send a request to delete the test survey after verification so as to not impact the client survey collection data and their statistics.
Example #2): For an e-commerce company, let’s assume a pricing update SQL job runs at midnight every day and uploads the finalized price to the website. We cannot run this SQL on-demand, multiple times for the purpose of post-release verification as this may cause unfinalized data to be pushed to production.
Moreover, it can increase the chances of DB deadlocks and high consumption of CPU and memory resources during peak business hours which can affect the client application performance.
Following changes need to be made to a social media application, specifically to the sign up process
Post production release verification Plan:
|1||Go to Livesiteurl||Website homepage should load successfully||Pass|
|2||Click on Sign up as new user||User should be redirected to the registration/sign up page||Pass|
|3||Fill in the required fields and click on Register button|
-Enter last name as ‘Lee’
-Toggle the privacy button to Do not Display
-Chose an Avatar
|-User should be redirected to their Profile page after successful registration. |
-User phone number should not be shown
-User selected Avatar should show
|Partial Pass||Avatar is not rendering properly and is showing as broken image. Reported in JIRA as BUG-1088|
|4||Monitoring - Verify if the application performance has improved after this release||Reduction of API calls during Sign up process should improve application performance||Ongoing||Action is on Dev Lead and Dev Ops team to monitor application for 24 hours|
|5||Post release cleanup||Delete the test account created||Done|
With most of the software companies now adopting the Agile methodology, the number of production releases has increased.
For example, while using Waterfall model, a team may have a production release every 1.5 months, however with the Agile process, the same team may now have a production release every 2-3 weeks.
With every production release, we have a possibility of knowingly or unknowingly impacting the functionality of the live clients. The adoption of post production release verification immediately after release can provide additional confidence on the release at the same time providing the safety net of rolling back the release before our live clients come across some issues.
For high impact/risk projects, post-production release verification plan can be structured based on the priority of the test scenario. Critical priority test can be executed first and communication sent to stakeholders about results and any issues. If no critical issues are found, then the post production release verification can continue, otherwise, the decision for roll back needs to be made to minimize application downtime and impact to live clients.
Additionally, post-production release testing can be automated and the test scripts can be run on demand after every release as a Regression test. Again, due care should be taken while running the automated test scripts on production as it may affect live client data and functionality.
Post production release verification is the last line of defense for any software company. If we do not catch the issues, our customers will and this can be devastating to the reputation of any Software company.
In order to maintain the reliability of the product, it is essential that we verify the changes deployed to production immediately after deployment.
About the author: This helpful article is written by Neha B. She is currently working as a Quality Assurance Manager and specialize in leading and managing In-house and Offshore QA teams.
Share your post-production release testing strategy / tips / experience with our readers.