How Do You Decide Which Defects are Acceptable for the Software to Go-live?

Software Go-Live is always a big event for any software product. It is important to absolutely make sure that everything works and that we are releasing quality software to the users.

A bad or premature or unstable or difficult to use product can cause a lot of losses financially and could also make the user lose trust in the brand itself.

Often times, we hear that testing should be done until we meet the exit criteria. We also hear that defects have to be fixed to an acceptable level. 

While, these are great sounding guidelines, they are vague.

To be more specific:

• What percentage of defects are acceptable for software to go-live?
• How do you decide on the open defects that the software can go-live with?
• What kinds of defects are more serious than the others?

Recommended read => When to stop Testing?

Have you had these questions ever? Then, this article is going to help you answer them. Read on…

Complex software is not defect free and it is a chicken and egg story on closing defects vis-à-vis working software.

The more you fix defects there is more likelihood that a new defect has been injected while closing the defect. So,

• How do you decide on the extent of defects and the type of defects that you can go-live with?
• How do you baseline the software to be deployed to go-live?
• How do UAT coordinators make the call to go-live or not?
• What parameters should software get judged against?
• How do we answer – Is the software fit for use and will it bring value to the stakeholders?

Going live into production is a major milestone for the customer as well as the Vendor as it is usually linked to payment milestones. Both of them have an equal responsibility in ensuring that the large transformational projects are successful.

My experience shows that customers want their value for money and have an exit criterion for UAT to go-live with.

The said exit criteria would more or less define the acceptable extent of problems in all areas of the application, such as:

1. Functional
2. Performance and Load
3. Usability
4. Security
5. Integration with external systems
6. Reports
7. Data migration

I believe every single one of these types of defects needs to be explained further. And, that is exactly what we will do now:

#1. Functional Defects:

If the software is created as per the specifications given by the customer, then it has to meet the requirements. Any deviations are logged as functional defects.

Functional defects are then classified according to severity and priority.

The following are important considerations:

• High severity and priority defects are usually the ones that would impact the day to day usage of the software. These types of defects are the ones that must be fixed before we go-live. No exceptions.
• Sometimes Functional defects are classified as Change Requests as they were not a part of the originally given requirements. Such CRs, that are a must for business to work after Go-live, are also a must to be implemented.
• Classification of defects and prioritization of functional defects are done by the UAT coordinators in collaboration with business users and Business analysts. Usually, the customer has an exit criteria of how much % of defects can be open for go-live.

#2. Performance and Load Defects:

Performance defects are important to consider for go-live and more so if the software is to be used by external users.

If the software is slow for a given number of users, the users would avoid using the software as it takes a lot of time to load up. Users tend to move to competitor’s site if the software is very slow thereby losing business.

Sometimes, the parts of the application that are not client facing can also impact performance.

For example:  If there is a batch process that runs at the end of every day and if the application’s response time suffers while this goes on, then the performance of the batch is also a factor to consider.

• Performance is usually measured in terms of response time of screens to render and become available to the users while there are a certain number of concurrent users on the system.
• Performance tests are done using tools such as LoadRunner, WebLoad, Neoload etc.
• The performance of the software at a given load and at a future predicted load is usually documented in the contract and has to be demonstrated before go-live.
• The screens or parts of the application that are used less by the users are deferred to evaluations after go-live.
• Performance is also dependent on the type of hardware and network conditions that the software is deployed on.
• Performance tests are done during UAT in the specified hardware using performance tools and their defects are tracked in a fashion similar to that of functional defects. They are also prioritized and a consensus is reached on meeting the exit criteria for go-live.
• Usually, performance and load tests in UAT are done after the functional UAT by the business users are completed and an acceptable exit criterion for functional defects is reached.

#3. Usability defects:

The software created should be easily usable by the end users using different hotkeys, shortcuts, the minimum number of screen navigation, pagination etc. The software must be smart and intuitive.

If there are too many movements of the page before moving to the appropriate screen, the users usually show less interest in using the software.

• Usability guidelines are created before the software is built. The software has to adhere to these guidelines.
• There may also be tool restrictions while creating the software which has to be overcome smartly before the software is usable by the end users.
• With highly usable software, an end user can input data as much as 5 times the regular software.
• The look and feel of the software has to be crisp and also legal issues have to be sorted out before go-live.
• Many times a usability consultant is appointed to ensure a smooth usability experience to the users.
• The documentation that has to go out with the software application also has to adhere to stringent usability guidelines as they can be used legally.
• The usability defects logged by UAT testers /external testers are also prioritized as functional and performance defects and have to meet the exit criteria for go-live.

#4. Security defects:

Security of the software is a hot issue as the software application can be hacked and customer sensitive data can be stolen within no time.

Hence, reliable software should not allow even a very competent hacker to get into the application without proper privileges.

• Security testing is done in UAT with specific inputs into the software to ensure that it is not hackable.
• Security testing is done by legal hackers who try to hack the software to check if it is vulnerable.
• All security defects have to be closed before the system goes live.
• Security also means Login and roles and privileges to various users (external and internal) to use different sections of the applications and also for creating and approving data.

#5. Integration with external software systems:

Usually, a software application that is to be deployed at the customer’s site has to interface with any existing software that might already exist there.

For example: With the printing system, they have in use or it could be external systems such as a billing application or data screen systems.  The software application being deployed should seamlessly integrate with these external systems. All input and output to these systems should be working synchronously. Technology today encompasses mobile apps and different software platforms that the application has to be compatible with.

Checking for external system interfacing should be extensively carried out during system and UAT stages. It should be a must on the exit criteria that should be satisfied before go-live.

#6. Reports:

Reports from the software application are a critical way to show that the data inside the application is tallying.

For example: all billing related data has to tally in the credit and debit balances.

• All data in the software must reconcile. This reconciliation of data within the software is shown through reports and they must be working as intended.
• This is especially true if data migration from an old system to a new system is the primary intention of the current release.

#7. Data migration:

If an old system is being replaced by a new one, data from the old system is moved to the new one (after a cut-off date is arrived at by using the new system). The data migrated should be supported by the new system as defined during requirement gathering.

All old data may not be available in the new system; however, a snapshot of the old data could be available in the new system. This data should be available as agreed upon.

Note: The above list is not exhaustive. Depending on the type of the application, there might be more things you have to validate or not everything above might be applicable. So, a thorough understanding of the software, the business purpose, the user expectations and architectural or hardware dependencies is a must to develop comprehensive exit criteria.

An example exit criteria for go-live:

This is just an example. It can vary from project to project.

• 100% of Priority 1 defects are closed (Severity Critical and priority 1)
• 90% of priority 2 defects are closed (Severity high and priority 2) with a logical workaround being available for the rest of the 10% of the defects. And, a plan is available for closing the rest of the 10% of the defects.
• Production deployment and sanity checklist is ready.
• Production support team has been formed and ready for closing tickets.
• 70% of priority 3 defects are closed and a plan is in place for closing rest of the 30% of low defects.

A few points to note:

• All severity and priority definitions are decided during the business meetings between the customer and vendor at the start of the program.
• After all the UAT defects are logged and all the other defects are being closed, UAT coordinators and Business sponsors meet to take stock of pending and open defects. If all defects that are required for Day-1 go-live are closed, the business sponsors see their readiness for go-live and get the software into production.

In conclusion

We hope this article has given you some insights as to some of the important consideration that goes into creating rock-solid exit criteria that shield the software from potential failures in productions.

About the author: This is a guest article by Krishnan Venkatraman. He is having nearly 18 years of experience in Software testing. He has worked on many large and complex software testing projects.

Feel free to post your queries/comments below.