Which Defects are Acceptable for Software Go-live

While we want to make sure that everyone is working well, defects can still occur. But how do you determine which defects are acceptable before the go-live? Let’s find out.

Software Go-Live is always a big event for any software product. It is important to make sure that everything works and that we are releasing quality software to our users.

A bad, premature, unstable or difficult to use product can cause a lot of losses financially and could also make the user lose trust in the brand itself.

Many times, we hear that testing should be done until we meet the exit criteria and that defects have to be fixed to an acceptable level.

While these are great sounding guidelines, they are vague.

Deciding which defects are acceptable

To be more specific:

• What percentage of defects are acceptable for software to go-live?
• How do you decide on the open defects that the software can go-live with?
• What kinds of defects are more serious than the others?

Recommended read => When to stop Testing?

Have you ever had these questions before? Then, this article is going to help you answer them. Read on…

Complex software is not defect-free, and it is a chicken and egg story on closing defects vis-à-vis working software.

The more you fix defects there is a greater likelihood that a new defect has been injected while closing the defect. So,

• How do you decide on the extent of defects and the type of defects that you can go-live with?
• How do you baseline the software to be deployed to go-live?
• How do UAT coordinators make the call go-live or not?
• What parameters should software be judged against?
• How do we answer – Is the software fit for use and will it bring value to the stakeholders?

Going live into production is a major milestone for the customer as well as the Vendor as it is usually linked to payment milestones. Both of them have an equal responsibility in ensuring that the large transformational projects are successful.

My experience shows that customers want their value for money and have an exit criteria for UAT to go-live with.

The said exit criteria would more or less define the acceptable extent of problems in all areas of the application, like the following:

1. Functional
2. Performance and Load
3. Usability
4. Security
5. Integration with external systems
6. Reports
7. Data migration

I believe every single one of these types of defects needs to be explained further. And this is exactly what we will do now:

#1. Functional Defects:

If the software is created as per the specifications given by the customer, then it has to meet the requirements. All deviations are logged as functional defects.

Functional defects are then classified according to severity and priority.

The following are important considerations:

• High severity and priority defects are usually the ones that would impact the day to day usage of the software. These types of defects are the ones that must be fixed before we go-live. No exceptions.
• Sometimes Functional defects are classified as Change Requests as they were not part of the originally given requirements. Such CRs, that are a must for businesses to work after Go-live, are also a must to be implemented.
• Classification of defects and prioritization of functional defects are done by UAT coordinators in collaboration with business users and business analysts. Usually, the customer has an exit criteria of how much % of defects can be open for go-live.

#2. Performance and Load Defects:

Performance defects are important to consider for go-live and more so if the software is to be used by external users.

If the software is slow for a given number of users, the users would avoid using the software as it takes a lot of time to load up. Users tend to move to a competitor’s site if the software is very slow thereby losing business.

Sometimes, the parts of the application that are not client facing can also impact performance.

For example: If there is a batch process that runs at the end of every day and if the application’s response time suffers while this goes on, then the performance of the batch is also a factor to consider.

• Performance is usually measured in terms of response time of screens to render and become available to the users while there are a certain number of concurrent users on the system.
• Performance tests are done using tools such as LoadRunner, WebLoad, Neoload etc.
• The performance of the software at a given load and at a future predicted load is usually documented in the contract and has to be demonstrated before go-live.
• Screens or parts of the application that are used less by the user are deferred to evaluations after go-live.
• Performance is also dependent on the type of hardware and network conditions that the software is deployed on.
• Performance tests are done during UAT in the specified hardware using performance tools and their defects are tracked in a fashion similar to that of functional defects. They are also prioritized and a consensus is reached on meeting the exit criteria for go-live.
• Usually, performance and load tests in UAT are done after the functional UAT by the business users are completed and an acceptable exit criterion for functional defects is reached.

#3. Usability defects:

The software created should be easily usable by the end users using different hotkeys, shortcuts, the minimum number of screen navigation, pagination etc. The software must be smart and intuitive.

If there are too many movements on the page before moving to the appropriate screen, the users usually show less interest in using the software.

• Usability guidelines are created before the software is built. The software has to adhere to these guidelines.
• There may also be tool restrictions while creating the software which has to be overcome smartly before the software is usable by the end users.
• With highly usable software, an end user can input data as much as 5 times the regular software.
• The look and feel of the software has to be crisp and also legal issues have to be sorted out before go-live.
• Often a usability consultant is appointed to ensure a smooth usability experience for the users.
• The documentation that has to go out with the software application also has to adhere to stringent usability guidelines as they can be used legally.
• The usability defects logged by UAT testers/external testers are also prioritized as functional and performance defects and have to meet the exit criteria for go-live.

#4. Security defects:

Security of the software is a hot issue as the software application can be hacked and customer sensitive data can be stolen within no time.

Hence, reliable software should not allow even a very competent hacker to get into the application without proper privileges.

• Security testing is done on UAT with specific inputs into the software to ensure that it is not hackable.
• Security testing is done by legal hackers who try to hack the software to check if it is vulnerable.
• All security defects have to be closed before the system goes live.
• Security also refers to Login and roles and privileges for various users (external and internal) to use different sections of the application and also to create and approve data.

#5. Integration with external software systems:

Usually, a software application that is to be deployed at the customer’s site has to interface with any existing software that might already exist there.

For example: With the printing system, they have in use or it could be external systems such as a billing application or data screen systems.  Software applications being deployed should seamlessly integrate with these external systems. All input and output to these systems should be working synchronously. Technology today encompasses mobile apps and different software platforms that the application has to be compatible with.

Checking for external system interfacing should be extensively carried out during system and UAT stages. The exit criteria should be a must before the go-live.

#6. Reports:

Reports from the software application are a critical way to show that the data inside the application is tallying.

For example: all billing related data has to tally in the credit and debit balances.

• All data in the software must be reconciled. This reconciliation of data within the software is shown through reports and they must be working as intended.
• This is especially true if data migration from an old system to a new system is the primary intention of the current release.

#7. Data migration:

If an old system is being replaced by a new one, data from the old system is moved to the new one (after a cut-off date is arrived by using the new system). The data migrated should be supported by the new system as defined during requirement gathering.

All old data may not be available in the new system; however, a snapshot of the old data could be available in the new system. This data should be available as agreed upon.

Note: The above list is not exhaustive. Depending on the type of the application, there might be a few more things you have to validate or not everything above might be applicable. Therefore, a thorough understanding of the software, the business purpose, the user expectations and architectural or hardware dependencies is a must to develop comprehensive exit criteria.

Example exit criteria for go-live:

This is just an example. It can vary from project to project.

• 100% of Priority 1 defects are closed (Severity Critical and priority 1)
• 90% of priority 2 defects are closed (Severity high and priority 2) with a logical workaround being available for the rest of the 10% of the defects. Plans are available for closing the rest of the 10% of the defects.
• Production deployment and sanity checklist is ready.
• Production support team has been formed and is ready to close tickets.
• 70% of priority 3 defects are closed and a plan is in place to close the rest of the 30% of low defects.

A few important points to note:

• All severity and priority definitions are decided during the business meetings between the customer and vendor at the start of the program.
• Once all UAT defects are logged and all other defects are being closed, UAT coordinators and business sponsors meet to take stock of the pending and open defects. If all defects that are required for Day-1 go-live are closed, the business sponsors will see their readiness for go-live and get the software into production.

In conclusion

We hope this article was able to provide some insights as to some of the important considerations that go into creating rock-solid exit criteria that shield the software from potential failures in production.

About the author: This is a guest article by Krishnan Venkatraman. He has nearly 18 years of experience in Software testing. He has worked on many large and complex software testing projects.

Feel free to post your queries/feedback below. We would love to hear from you.