When to Stop Testing (Exit Criteria in Software Testing)

Exit criteria in Testing:

“Well begun is half done” – Applies everywhere, even software testing.

Often we see software testers very enthusiastic at the beginning of the project. We create testing documents such as Test Strategy, Test Plan or Test Cases eagerly and enthusiastically.

Then we get to testing software with a BANG! This is only amplified by the interesting defects we find at the beginning of the project. Getting them resolved will only add to our accomplishment. 

As we find loads of defects and complete the first run we move on to the next phase. When we get to the second run we kind of relax and as is the general human tendency of getting bored with testing the same thing in the second run.

Many testers feel that it becomes monotonous work in later runs and start losing interest in testing the same software over and over again. When we reach to, maybe the third run, one question starts haunting us and that is “When to Stop Testing the software?”

I bet you must have felt the same way and asked, “When to stop testing?”, at least once. I would say the question is “When, where and how to stop Testing?” :)

Conceptually we have read and many testers believe that there cannot be a specific condition or equation to decide “When to stop testing?” There are a number of factors to consider before we conclude on this question.

In today’s article, I would like to share my thoughts on how to conclude testing activities when we reach to a point in our testing cycle where we can say this testing is enough. We will do this with the help of a few real life examples in a typical testing cycle.

When is it Enough Testing?

When can we say that this much testing is enough? Can testing ever be completed?

In order to answer these questions, we will have to analyze testing activities from start to end. Please note that – I am going to define these activities in a lay man’s terms – Not in a fancy technical way.

Let’s consider you are beginning testing of a new project.

Initial Activities:

• The testing team receives requirements.
• The testing team starts planning and designing.
• Initial Test documents are ready and reviewed.

Testing Run #1)

• The testing team starts test execution once they receive the developed product.
• During the testing phase, they execute various scenarios in order to break the software and find many defects. (Also, the defect rate here is higher because the application is new and is undergoing evaluation for the very first time.)
• The Defects get fixed by developers and returned back to test team for retest.
• The testing team performs retesting of the defects and executes regression.
• Once most of the high severity defects are resolved and the software looks stable, development team releases the next version.

Testing Run #2)

• The testing team starts the second run of testing and similar activities are executed as Run 1.
• In this process during the second testing run, few more defects are caught.
• The Defects get fixed by developers and returned back to the test team for a retest.
• Testing team re-tests the defects and performs regression.

This can continue forever. Run 3, Run 4 onwards until all defects in the software are found and software becomes bug-free.

If we want to draw a flow chart for these activities, it will look roughly like below:

From the above flow chart, we can clearly conclude that testing can continue until all defects in the software are found.

But the question is – Is it possible to find every single defect in the software? Let’s try to find the answer for this million dollar question :).

Stopping when all defects are found: Is it possible?

Most software is complex and has an enormous testing scope. It is not impossible to find all defects in the software but it will take forever.

Even after finding many bugs in the software, no one can actually guarantee that the software is defect free now. There cannot be a situation where we can confidently say that we have completed testing, found all defects in the software and it does not have any more bugs.

Moreover, the purpose of testing is not to find each and every defect in the software. The intent of software testing is to prove that the software does work as intended by breaking it or finding deviation between its current behavior and expected behavior.

There are unlimited defects in software and hence it’s impractical to test it until all defects are found as we can never know which defect is the last one. The truth is we cannot depend on finding all the defects in the software to conclude our testing.

Honestly speaking, testing is endless and testing cycles will continue until a decision is made when and where to stop. Now it becomes even more complicated to come to a decision to stop testing. If “stopping when all defects are found” is not the criterion to stop testing then on what basis should it be decided?

Decision to stop testing: Exit criteria

Let’s now try to understand – What are the most important factors to be considered while concluding testing activities? I feel the decision to stop testing is mostly dependent on Time, Budget and Extent of Testing.

The most common approach is to stop when either Time / Budget is exhausted or all test scenarios are executed. However, with this approach, we will be compromising on the quality of testing and this will not give enough confidence about the software; how?

Let’s see with an example.

Test Scenario:

Suppose you are testing a software module.  You have been allocated certain budget to cover it. The project timelines are for a month. Total Test Scenarios are 200. You are the only one testing this module.

Scenario #1)

Week 1: You find the showstopper / severity 1 defect on day 1 and the entire testing is blocked for 3 days. Hence you are not able to execute any of the scenarios until Severity 1 defect is resolved. After losing 3 days, the blocker is resolved and you continue with your execution.

At the end of the week, you complete 20 scenarios with few more important high priority defects.

Week 2: You start testing the software putting more focus on defects finding. You open few more Severity 1, Severity 2 and Severity 3 defects during the second week and at the end of the week, you are able to cover 70 Scenarios.

Week 3: By the start of the 3^rd week you get all the high priority defects resolved so along with execution of pending scenarios you now have to re-test all the defects which have landed in the testing bucket. Continuing with the good progress you cover 120 scenarios with additional defects.

By this time all high priority defects are already found and reported. So, now you have only Severity 3 defects left to be found.

Week 4:  By week 4 you need to re-test most of the opened defects and remaining 80 Scenarios. With this by the end of week 4, you are able to complete up to 180 scenarios with all High and Medium priority defects fixed and re-tested.

Putting this information in Tabular form:

Should you stop here?

The reason that you have exhausted Testing Time completely and have reported and fixed most of the high priority defects. Will stopping at this point give you confidence about the software? Not really due to below reasons:

• Scenarios are not executed completely.
• Few flows are not even tested once.
• All the Scenarios which are covered are executed only once.
• Software still has defects in it.
• Regression is not covered.

Scenario #2)

 Week 1: You find Severity 1 defect on day 1 and complete testing is blocked for 3 days. Hence you are not able to execute any of the scenarios until Severity 1 Defect is resolved. After losing 3 days the blocker is resolved and you continue with your execution.

At the end of the week, you complete 20 scenarios with few more defects. This week remains same as Scenario 1.

Week 2: You open few more Severity 1, Severity 2 and Severity 3 defects during the second week, but the focus is to cover more scenarios to cover backlog from week 1. At the end of the week, you are able to cover 120 Scenarios.

Week 3: By the start of the 3^rd week you get all the open defects resolved so along with execution of pending scenarios you now have to re-test all the defects which are landed in a testing bucket. Still continuing with good progress at the end the no of scenarios completed becomes 200 with additional defects.

Now you can only report Severity 2 and Severity 3 defects.

Putting this information in Tabular form:

Should you stop here?

You have covered all Testing scenarios completely once and have opened few major defects. Will stopping at this point give you confidence about the software?

Not really due to below reasons:

• All Scenarios are executed only once.
• Software still has many major defects in it.
• Regression is not covered.

We can see that the quality is compromised in above both the scenarios. The best approach will be finding a point where all the factors from scenario 1 and scenario 2 are covered and quality is also not compromised. To achieve this we will have to define certain criteria at the beginning of testing. 

These criteria are termed as Completion or Exit Criteria. It is the answer to our question – “When to stop Testing?”.

What is Completion or Exit Criteria?

The exit criteria get evaluated at the end of the testing cycle and is defined in Test Plan. It is the set of conditions or activities which must be fulfilled in order to conclude testing.

The Exit criteria define how much testing is enough and when testing activities can be declared complete. Coverage and completion criteria are combined to define exit criteria for testing.

What should be present in Exit Criteria?

Ideally, Exit or Stop Criteria is defined by combining various factors and hence is unique across all projects. It depends on the project requirement and hence should be defined during Test Planning; at the beginning of the project. Parameters defined in it should be quantified as much as possible.

Below are few pointers to be considered while defining Exit Criteria in case of Functional or System Testing. You may combine few or all the below factors while deciding where to stop testing as per your project needs.

Testing can be stopped when:

Requirements:

• 100% Requirements coverage is achieved.

Defects:

• Defined / Desired Defect count is reached.
• All Show Stopper defects or Blockers are fixed and No known Critical / Severity 1 defect is in Open Status.
• All High Priority defects are identified and fixed.
• Defect Rate falls below defined acceptable rate.
• Very few Medium Priority defects are open and have a workaround in place.
• Very few low priority open defects that do not impact software usage.
• All High Priority defects are re-tested and closed and corresponding Regression scenarios are successfully executed.

 Test Coverage:

• Test Coverage should be 95% achieved.
• Test case Pass Rate should be 95%. This can be calculated by formula
  • ( Total No of TCs Passed / Total number of TCs ) * 100.
• All critical Test cases are passed.
• 5% Test cases can be failed but the Failed Test cases are of low priority.
• Complete Functional Coverage is achieved.
• All major functional / business flows are executed successfully with various inputs and are working fine.

Deadlines:

• Project Deadline or Test Finish deadline is reached.

Test Documents:

• All Test Documents / deliverables (Example – Test Summary Report) are prepared, reviewed and published across.

Budget:

• Complete Testing Budget is exhausted.

“Go / No Go” Meetings:

• “Go / No Go” meeting has been conducted with stakeholders and a decision is made whether the project should go to production or not.

Conclusion:

Let’s make it very simple at the end.

Please answer questions with a simple Yes or No.

If you get most of the answers as Yes that means you can stop testing at this point. If most of the answers are No that means you must check what is missing from testing and this may help you find an escaping production defect :)

• Are all test cases executed at least once?
• Is the Test Case Pass rate as defined?
• Is complete test coverage achieved?
• Are all functional / Business flows executed at least once?
• Is the decided defect count reached?
• Are all Major High Priority Defects fixed and closed?
• Have all Defects been Retested and closed?
• Has Regression been done for all open defects?
• Have you exhausted the testing budget?
• Has the Testing end time reached?
• Are all Test Deliverables reviewed and published?

About the author:  This is a guest article by Renuka K. She is having 11+ years of Software testing experience.

Hope this article was helpful to you. I would also like to hear your thoughts / comments on the topic.

Happy Testing!