How Do You Decide Which Defects are Acceptable for the Software to Go-live?

Software Go-Live is always a big event for any software product. It is important to absolutely make sure that everything works and that we are releasing quality software to the users.

A bad or premature or unstable or difficult to use product can cause a lot of losses financially and could also make the user lose trust in the brand itself.

Often times, we hear that testing should be done until we meet the exit criteria. We also hear that defects have to be fixed to an acceptable level. 

While, these are great sounding guidelines, they are vague.

To be more specific:

Recommended read => When to stop Testing?

Have you had these questions ever? Then, this article is going to help you answer them. Read on…

Complex software is not defect free and it is a chicken and egg story on closing defects vis-à-vis working software.

The more you fix defects there is more likelihood that a new defect has been injected while closing the defect. So,

Going live into production is a major milestone for the customer as well as the Vendor as it is usually linked to payment milestones. Both of them have an equal responsibility in ensuring that the large transformational projects are successful.

My experience shows that customers want their value for money and have an exit criterion for UAT to go-live with.

The said exit criteria would more or less define the acceptable extent of problems in all areas of the application, such as:

  1. Functional
  2. Performance and Load
  3. Usability
  4. Security
  5. Integration with external systems
  6. Reports
  7. Data migration

I believe every single one of these types of defects needs to be explained further. And, that is exactly what we will do now:

#1. Functional Defects:

If the software is created as per the specifications given by the customer, then it has to meet the requirements. Any deviations are logged as functional defects.

Functional defects are then classified according to severity and priority.

The following are important considerations:

#2. Performance and Load Defects:

Performance defects are important to consider for go-live and more so if the software is to be used by external users.

If the software is slow for a given number of users, the users would avoid using the software as it takes a lot of time to load up. Users tend to move to competitor’s site if the software is very slow thereby losing business.

Sometimes, the parts of the application that are not client facing can also impact performance.

For example:  If there is a batch process that runs at the end of every day and if the application’s response time suffers while this goes on, then the performance of the batch is also a factor to consider.

#3. Usability defects:

The software created should be easily usable by the end users using different hotkeys, shortcuts, the minimum number of screen navigation, pagination etc. The software must be smart and intuitive.

If there are too many movements of the page before moving to the appropriate screen, the users usually show less interest in using the software.

#4. Security defects:

Security of the software is a hot issue as the software application can be hacked and customer sensitive data can be stolen within no time.

Hence, reliable software should not allow even a very competent hacker to get into the application without proper privileges.

#5. Integration with external software systems:

Usually, a software application that is to be deployed at the customer’s site has to interface with any existing software that might already exist there.

For example: With the printing system, they have in use or it could be external systems such as a billing application or data screen systems.  The software application being deployed should seamlessly integrate with these external systems. All input and output to these systems should be working synchronously. Technology today encompasses mobile apps and different software platforms that the application has to be compatible with.

Checking for external system interfacing should be extensively carried out during system and UAT stages. It should be a must on the exit criteria that should be satisfied before go-live.

#6. Reports:

Reports from the software application are a critical way to show that the data inside the application is tallying.

For example: all billing related data has to tally in the credit and debit balances.

#7. Data migration:

If an old system is being replaced by a new one, data from the old system is moved to the new one (after a cut-off date is arrived at by using the new system). The data migrated should be supported by the new system as defined during requirement gathering.

All old data may not be available in the new system; however, a snapshot of the old data could be available in the new system. This data should be available as agreed upon.

Note: The above list is not exhaustive. Depending on the type of the application, there might be more things you have to validate or not everything above might be applicable. So, a thorough understanding of the software, the business purpose, the user expectations and architectural or hardware dependencies is a must to develop comprehensive exit criteria.

An example exit criteria for go-live:

This is just an example. It can vary from project to project.

A few points to note:

In conclusion

We hope this article has given you some insights as to some of the important consideration that goes into creating rock-solid exit criteria that shield the software from potential failures in productions.

About the author: This is a guest article by Krishnan Venkatraman. He is having nearly 18 years of experience in Software testing. He has worked on many large and complex software testing projects.

Feel free to post your queries/comments below.