When to Stop Testing? Testing Exit Criteria Decision

In this article, we have provided a detailed overview of the Exit criteria for Testing. Let’s begin.

“Well begun is half done” – This adage applies in every field, even in the field of software testing.

It is often seen that we software testers are very enthusiastic at the beginning of the project. We are eager to create testing documents such as Test Strategy, Test Plan and Test Cases.

Then we get to testing software with a BANG! This is only amplified by the interesting defects we find at the beginning of the project. Getting them resolved will then add to our accomplishment.

As we find loads of defects and complete the first run we move on to the next phase. In the second run we tend to take it easy. It is a general human tendency of getting bored with testing the same thing in the second run.

Many testers feel that it becomes monotonous in later runs and start losing interest in testing the same software over and over again. By the time we reach to, maybe the third run, one question starts haunting us and that is “When to Stop Testing the software?”

When to Stop Testing

I bet you must have felt the same way and asked, “When to stop testing?”, at least once in your career. I would say the question is “When, where and how to stop Testing?” 🙂

Conceptually, we have read and many testers believe that there cannot be a specific condition or equation to decide “When to stop testing?” There are a number of factors to consider before we conclude on this question.

In today’s article, I would like to share my thoughts on how to conclude testing activities when we reach a point in our testing cycle where we can say this testing is enough. We will do this with the help of a few real life examples in a typical testing cycle.

When Is It Enough Testing

When can we say that this much testing is enough? Can testing ever be completed?

In order to answer these questions, we will have to analyze testing activities from start to finish. Please note that I am going to define these activities in layman’s terms and not in a complex technical way.

Please consider that you are beginning testing on a new project.

Initial Activities:

• The testing team has received the requirements.
• The testing team has started planning and designing.
• Initial Test documents are ready and reviewed.

Testing Run #1)

• The testing team will start test execution once they receive the developed product.
• During the testing phase, they execute various scenarios in order to break the software and find many defects. (Also, the defect rate here is higher because the application is new and is undergoing evaluation for the very first time.)
• Defects get fixed by the developers and returned back to the testing team for retest.
• The testing team performs retesting of the defects and executes regression.
• Once most of the high severity defects are resolved and the software looks stable, the development team releases the next version.

Testing Run #2)

• The testing team started the second run of testing and similar activities were executed as Run 1.
• During the second testing run, a few more defects were caught in this process.
• Defects get fixed by developers and returned back to the test team for a retest.
• Testing team re-tests the defects and performs regression.

This can continue forever. Run 3, Run 4 onwards until all defects in the software are found and software becomes bug-free.

If we want to draw a flow chart for these activities, it will look roughly like below:

From the above flow chart, we can clearly conclude that testing can continue until all defects in the software are found.

But the question is – Is it possible to find every single defect in the software? Let’s try to find the answer for this million dollar question :).

Stop when all defects are found: Is it possible

Most software is complex and has an enormous testing scope. It is not impossible to find all defects in the software but it will take forever.

Even after finding many bugs in the software, no one can guarantee that the software is defect free now. There cannot be a situation where we can confidently say that we have completed testing, found all defects in the software and it does not have any more bugs.

Moreover, the purpose of testing is not to find each and every defect in the software. The intent of software testing is to prove that the software does work as intended by breaking it or finding deviation between its current behavior and expected behavior.

There are unlimited defects in software and hence it’s impractical to test it until all defects are found as we can never know which defect is the last one. The truth is we cannot depend on finding all the defects in the software to conclude our testing.

Honestly speaking, testing is endless and testing cycles will continue until a decision is made on when and where to stop. Now it has become even more complicated to come to a decision to stop testing. If “stopping when all defects are found” is not the criterion to stop testing then on what basis should it be decided?

Decision to stop testing: Exit criteria

Let’s now try to understand – What are the most important factors to be considered while concluding testing activities? I feel the decision to stop testing is mostly dependent on Time, Budget and Extent of Testing.

The most common approach is to stop when either Time / Budget is exhausted or all test scenarios are executed. However, with this approach, we will be compromising on the quality of testing and this will not give enough confidence about the software; how?

Let’s see with an example.

Test Scenario:

Suppose you are testing a software module.  You have been allocated a certain budget to cover it. Here are the project timelines for the month. Total Test Scenarios are 200. You are the only one testing this module.

Scenario #1)

Week 1: You find the showstopper / severity 1 defect on day 1 and the entire testing is blocked for 3 days. Hence, you will not be able to execute any of the scenarios until the Severity 1 defect is resolved. After missing 3 days, the blocker is resolved and you continue with your execution.

At the end of the week, you will complete 20 scenarios with a few more important high priority defects.

Week 2: You start testing the software by putting more focus on defects findings. You open a few more Severity 1, Severity 2 and Severity 3 defects during the second week and at the end of the week, you are able to cover 70 Scenarios.

Week 3: By the start of the 3^rd week you get all the high priority defects resolved so along with execution of pending scenarios you now have to re-test all the defects which have landed in the testing bucket. Continuing with the good progress you cover 120 scenarios with additional defects.

At this time, all high priority defects have already been found and reported. So now you have only Severity 3 defects left to be found.

Week 4: By week 4 you need to re-test most of the open defects and the remaining 80 Scenarios. With this, by the end of week 4, you will be able to complete up to 180 scenarios with all high and medium priority defects fixed and re-tested.

Putting this information in Tabular form:

Should we stop here?

You have exhausted the Testing Time completely and have reported and fixed most of the high priority defects. Will stopping at this point give you confidence about the software? Not really, due to the following reasons:

• Scenarios are not executed completely.
• Few flows are not even tested once.
• All the Scenarios which are covered are executed only once.
• Software still has defects in it.
• Regression is not covered.

Scenario #2)

 Week 1: You will find Severity 1 defect on day 1 and complete testing is blocked for 3 days. Hence, you will not be able to execute any of the scenarios until the Severity 1 Defect is resolved. After missing 3 days, the blocker is resolved and you continue with your execution.

By the end of the week, you will complete 20 scenarios with a few more defects. This week remains the same as Scenario 1.

Week 2: You open a few more Severity 1, Severity 2 and Severity 3 defects during the second week, but the focus is to cover more scenarios to cover backlogs from week 1. By the end of the week, you will be able to cover 120 Scenarios.

Week 3: By the start of the 3^rd week you get all the open defects resolved, so along with the execution of the pending scenarios you now have to re-test all the defects that have landed in the testing bucket. Still continuing with good progress at the end the no of scenarios completed becomes 200 with additional defects.

You can now only report Severity 2 and Severity 3 defects.

Putting this information in Tabular form:

Should we stop here?

You have covered all the Testing scenarios completely once and have opened a few major defects. Will stopping at this point give you confidence about the software?

Not really, due to the following reasons:

• All Scenarios are executed only once.
• Software still has many major defects in it.
• Regression is not covered.

We can see that the quality is compromised in both the scenarios. The best approach will be to find a point where all the factors from scenario 1 and scenario 2 are covered and quality is also not compromised. To achieve this we will have to define certain criteria at the beginning of testing. 

These criteria are termed as Completion or Exit Criteria. It is the answer to our question – “When to stop Testing?”.

What is the Completion and Exit Criteria

The exit criteria gets evaluated at the end of the testing cycle and is defined in the Test Plan. It is the set of conditions or activities which must be fulfilled in order to conclude the testing process.

Exit criteria defines how much testing is enough and when testing activities can be declared complete. Coverage and completion criteria are combined to define exit criteria for testing.

What should be present in Exit Criteria

Ideally, Exit or Stop Criteria is defined by combining various factors and hence is unique across all projects. It depends on the project requirements and hence should be defined during Test Planning; at the beginning of the project. Parameters defined in it should be quantified as much as possible.

Below are a few pointers to be considered while defining Exit Criteria in case of Functional or System Testing. You may combine a few or all of the below factors while deciding where to stop testing as per your project needs.

Testing can be stopped when:

Requirements:

• 100% Requirements coverage has been achieved.

Defects:

• Defined / Desired Defect count reached.
• All Show Stopper defects and Blockers are fixed and no known Critical / Severity 1 defect is in Open Status.
• All High Priority defects are identified and fixed.
• Defect Rate falls below the defined acceptable rate.
• Very few Medium Priority defects are open and have a workaround in place.
• Very few low priority open defects that do not impact software usage.
• All High Priority defects are re-tested and closed and corresponding Regression scenarios are successfully executed.

 Test Coverage:

• Test Coverage should be 95% achieved.
• Test case Pass Rate should be 95%. This can be calculated by formula
  • ( Total No of TCs Passed / Total number of TCs ) * 100.
• All critical Test cases have passed.
• 5% Test cases can be failed but the Failed Test cases are of low priority.
• Complete Functional Coverage has been achieved.
• All major functional / business flows have been executed successfully with various inputs and are working fine.

Deadlines:

• Project Deadline and Test Finish deadline have been reached.

Test Documents:

• All Test Documents / Deliverables (Example – Test Summary Report) are prepared, reviewed and published across.

Budget:

• The Complete Testing Budget has been exhausted.

“Go / No Go” Meetings:

• “Go / No Go” meeting has been conducted with stakeholders and a decision is made whether the project should go to production or not.

Conclusion:

Let’s make it very simple for you at the end. Please answer the questions with a simple Yes or No.

If you get most of the answers as yes, that means you can stop testing at this point. In case most of the answers are No, that means you must check what is missing from testing and this may help you find an escape production defect 🙂

• Have all test cases been executed at least once?
• Is the Test Case Pass rate as defined?
• Is complete test coverage achieved?
• Are all functional / Business flows executed at least once?
• Has the decided defect count been reached?
• Are all Major High Priority Defects fixed and closed?
• Have all the Defects been retested and closed?
• Has Regression been done for all open defects?
• Have you exhausted the testing budget?
• Has the Testing end time reached?
• Are all Test Deliverables reviewed and published?

About the author: This is a guest article by Renuka K. She is having 11+ years of Software testing experience.

Hope this article was helpful to you. I would also like to hear your thoughts / queries/feedback on the topic in the comments section below. 

Happy Testing!