In today’s article, we will learn with examples and graphs – Software Test Metrics and Measurements and how to use these in the Software Testing process.
There is a famous statement: “We can’t control things that we can’t measure”.
In software projects, measuring the quality, cost, and effectiveness of the project and the processes is most important. Without measuring these, a project can’t be completed successfully.
Here controlling the projects means how a project manager/lead can identify the deviations from the test plan ASAP to react in the perfect time. Generating test metrics based on the project needs is very important to achieve the quality of the software being tested.
Table of Contents:
Software Test Metrics and Measurements
What is Software Testing Metrics
A Metric is a quantitative measure of the degree to which a system, system component, or process possesses a given attribute.
Metrics can be defined as “STANDARDS OF MEASUREMENT”.
Software Metrics are used to measure the quality of the project. A metric is a unit used for describing an attribute. Metric is a scale for measurement.
Suppose “Kilogram” is a metric for measuring the attribute “Weight”. Similarly, in software, “How many issues are found in a thousand lines of code?”, here the number of issues is one measurement & No. of lines of code is another measurement. Metric is defined from these two measurements.
Test metrics example:
- How many defects exist within the module?
- How many test cases are executed per person?
- What is the Test coverage %?
What is Software Test Measurement
Measurement indicates the extent, amount, dimension, capacity, or size of some attribute of a product or process.
Test Measurement example: Total number of defects.
Please refer below diagram for a clear understanding of the difference between Measurement & Metrics.
Why Test Metrics
Generation of Software Test Metrics is the most important responsibility of the Software Test Lead/Manager.
Test Metrics are used to,
- Decide for the next phase of activities, such as estimating the cost & schedule of future projects.
- Understand the kind of improvement required to succeed in the project.
- Decide on the Process or Technology to be modified, etc.
Importance of Software Testing Metrics:
As explained above, Test Metrics are the most important to measure the quality of the software.
Now, how can we measure the quality of the software by using Metrics?
Suppose, if a project does not have any metrics, then how will the quality of the work done by a Test Analyst will be measured?
For Example, A Test Analyst has to,
- Design the test cases for 5 requirements.
- Execute the designed test cases.
- Log the defects & need to fail the related test cases.
- After the defect is resolved, we need to re-test the defect & re-execute the corresponding failed test case.
In the above scenario, if metrics are not followed, then the work completed by the test analyst will be subjective, i.e. the Test Report will not have the proper information to know the status of his work/project.
If Metrics are involved in the project, then the exact status of his/her work with proper numbers/data can be published.
i.e. in the Test Report, we can publish:
- How many test cases have been designed per requirement?
- How many test cases still need to be designed?
- How many test cases are executed?
- How many test cases are passed/failed/blocked?
- How many test cases have not been executed yet?
- How many defects are identified & what is the severity of those defects?
- How many test cases are failed due to one particular defect? etc.
Based on the project needs, we can have more metrics than the above list, to know the status of the project in detail.
The Test Lead/Manager will gain insight into the key points mentioned below based on the provided metrics.
- Percentage of work completed.
- Percentage of remaining work.
- Time to complete the remaining work.
- Whether the project going as per the schedule or lagging? etc.
If the project does not meet the timeline, the manager will inform the client and other stakeholders, providing reasons for the delay in preventing any last-minute surprises.
Metrics Life Cycle
Types of Manual Test Metrics
Testing Metrics are mainly divided into 2 categories.
- Base Metrics
- Calculated Metrics
Base Metrics: Base Metrics are the Metrics that are derived from the data gathered by the Test Analyst during the test case development and execution.
This data will be tracked throughout the Test Lifecycle. I.e. collecting the data like the Total no. of test cases developed for a project (or) no. of test cases that need to be executed (or) no. of test cases passed/failed/blocked, etc.
Calculated Metrics: Calculated Metrics are derived from the data gathered in Base Metrics. These Metrics are tracked by the test lead/manager for Test Reporting purposes.
Examples of Software Testing Metrics
Let’s take an example to calculate various test metrics used in software test reports:
Below is the table format for the data retrieved from the Test Analyst who is involved in testing:
Definitions and Formulas for Calculating Metrics:
#1) Test cases executed (%ge): This metric determines the percentage of test cases that have been executed.
%ge Test cases Executed = (No. of Test cases executed / Total no. of Test cases written) * 100.
So, from the above data,
%ge Test cases Executed = (65 / 100) * 100 = 65%
#2) Test cases not executed (%ge): This metric is used to obtain the pending execution status of the test cases in terms of %ge.
%ge Test cases not executed = (No. of Test cases not executed / Total no. of Test cases written) * 100.
So, from the above data,
%ge Test cases Blocked = (35 / 100) * 100 = 35%
#3) Test cases Passed (%ge): This metric is used to obtain the Pass %ge of the executed test cases.
%ge Test cases Passed = (No. of Test cases Passed / Total no. of Test cases Executed) * 100.
So, from the above data,
%ge Test cases Passed = (30 / 65) * 100 = 46%
#4) Test cases Failed (%ge): This metric is used to obtain the Fail %ge of the executed test cases.
%ge Test cases Failed = (No. of Test cases Failed / Total no. of Test cases Executed) * 100.
So, from the above data,
%ge Test cases Passed = (26 / 65) * 100 = 40%
#5) Test cases Blocked: This metric is used to obtain the blocked %ge of the executed test cases. A detailed report can be submitted by specifying the actual reason for blocking the test cases.
%ge Test cases Blocked = (No. of Test cases Blocked / Total no. of Test cases Executed) * 100.
So, from the above data,
%ge Test cases Blocked = (9 / 65) * 100 = 14%
#6) Defect Density = No. of Defects identified / size
(Here “Size” is considered a requirement. Hence, here the Defect Density is calculated as several defects identified per requirement. Similarly, Defect Density can be calculated as some Defects identified per 100 lines of code [OR] No. of defects identified per module, etc.)
So, from the above data,
Defect Density = (30 / 5) = 6
#7) Defect Removal Efficiency (DRE) = (No. of Defects found during QA testing / (No. of Defects found during QA testing + No. of Defects found by End-user)) * 100
DRE is used to identify the test effectiveness of the system.
Suppose, During Development & QA testing, we have identified 100 defects.
After the QA testing, during Alpha & Beta testing, the end-user / client identified 40 defects, which could have been identified during the QA testing phase.
Now, The DRE will be calculated as,
DRE = [100 / (100 + 40)] * 100 = [100 /140] * 100 = 71%
#8) Defect Leakage: Defect Leakage is the Metric that is used to identify the efficiency of the QA testing i.e., how many defects are missed/slipped during the QA testing.
Defect Leakage = (No. of Defects found in UAT / No. of Defects found in QA testing.) * 100
Suppose, During Development & QA testing, we have identified 100 defects.
After the QA testing, during Alpha & Beta testing, the end-user / client identified 40 defects, which could have been identified during the QA testing phase.
Defect Leakage = (40 /100) * 100 = 40%
#9) Defects by Priority: This metric is used to identify the no. of defects identified based on the Severity / Priority of the defect which is used to decide the quality of the software.
%ge Critical Defects = No. of Critical Defects identified / Total no. of Defects identified * 100
From the data available in the above table,
%ge Critical Defects = 6/ 30 * 100 = 20%
%ge High Defects = No. of High Defects identified / Total no. of Defects identified * 100
From the data available in the above table,
%ge High Defects = 10/ 30 * 100 = 33.33%
%ge Medium Defects = No. of Medium Defects identified / Total no. of Defects identified * 100
From the data available in the above table,
%ge Medium Defects = 6/ 30 * 100 = 20%
%ge Low Defects = No. of Low Defects identified / Total no. of Defects identified * 100
From the data available in the above table,
%ge Low Defects = 8/ 30 * 100 = 27%
Recommended reading => How to Write an Effective Test Summary Report
Conclusion
The metrics provided in this article are majorly used for generating the Daily/Weekly Status report with accurate data during the test case development/execution phase & this is also useful for tracking the project status & Quality of the software.
About the author: This is a guest post by Anuradha K. She has 7+ years of software testing experience and currently working as a consultant for an MNC. She also has good knowledge of mobile automation testing.
Which other test metrics do you use in your project? As usual, let us know your thoughts/queries in the comments below.
Was this helpful?
Recommended Reading
-
This Tutorial Explains The Role of Metadata in ETL, Examples & Types of Metadata, Metadata Repository & Challenges in Metadata Management: Data Mart in ETL was explained in detail in our previous tutorial. The concept of Metadata is very important in ETL and this tutorial will explain all about Metadata.…
-
We are working our way through figuring out how to derive an Automation Framework that works best for a certain testing project and also defining certain frameworks that already exist. The example that we were using in the previous QTP framework article was creating a new Gmail account. To start…
-
Software Testing: In this tutorial, we discuss the Evolution of Software Testing, the Software Testing Life Cycle, and the various phases involved in STLC. 8 Phases of Software Testing Life Cycle (STLC) Evolution: 1960’s Trend: 1990’s Trend 2000’s Trend: The trend and competency of testing are changing. Testers are now…
-
Software Testing Course Syllabus and Training Plan Week 1 Brief introduction to software systems and SDLC Basic concepts Basic Testing Vocabulary Quality Assurance versus Quality Control The Cost of Quality Software Quality Factors How Quality is Defined Why Do We Test Software? What is a Defect? The Multiple Roles of…
Thanks for your post.
There are more metrics used in our project. I am listing few of them,
Requirement/Usecase Coverage
Test case/Script design productivity
Test case/Script execution productivity
Test Effectiveness
Cost Of Quality (COQ)
Effort Variance
(Overall and Variance at each milestone)
Schedule variance
can you share a formula or how to calculate Test design productivity and test execution productivity?
1+0=0 is P2
1*0=1 is P1
4+5=9 is P4
4*1=41 is P3
@ All
Does any one have a sample excel sheet where we show all the defect details and test case reports in the call or to the client in the form of Graph or Pie chart( with formula based in excel sheet).If any please share
Easy understand nice good example
Thankyou
Hi Jordan,
Please send the excel template.
Thanks.
Debbie
Hi,
It’s something i really want to learn. But at present i am leading a project where test cases creation and execution were not used.
So please suggest me how to create test metrics based on number issue found and number of issues closed/reopened.
Thanks
how to give severity and priority for the below equations
1+0=0
1*0=1
4+5=9
4*1=41
Good one on types of metrics with examples.
Hi Jordan could you pleae send me the excel template of this presentation to be able just to apply my numbers and get the results.
my mail id is – sopan@gibots.com
7709914753
Its really good material, if you have few more metrics then its excellent like test coverage, defect aging
Defect leakage should be number of defects found in UAT/total number of defects(UAT+QA testing)*100..
In my opinion, difference between metrics and measurements is described quite messy in the article.
“Suppose, in general, “Kilogram” is a metric for measuring the attribute “Weight””
–Kilogram is rather measurement, than metric (which is also shown in Measurement & Metrics diagram, where Km and Hours are measurements).
All 13 testing metrics in example table are rather measurements.
If I consider the re-execution of the test cases due to failure rate, how does the calculations go for these metrics, especially when you are calculating the
– %age of Test Cases Failed or Passed
If i have executed 100% cases and all my test cases Pass, do we deliver and tell customer that my pass percent is 100%
Just came across this post, pertaining to comments..
@rajeshri : i have been working for UHG in bangalore, Its a brilliant company to work(Fortune14). They provide you with plenty of WFH. I have been working as QA TL, I used to work from my home(Chennai) 2weeks everymonth.
Hi Jordan,
Please send excel template
@Dipankar: It is obvious. DRE should be higher, it should be nearly 100%. If it is 100% then it represents testing is carried our perfectly hence there’s no UAT defects/Customer issues.
DRE should be higher and Defect Density should be lower(viz good for entire project)
What is the difference between test metric & traceability matrix?
The difference between metrics and measurement is simply. Metrics are mathematical (quantifiable), and can have a number applied to them. Measurement is the larger body, that contains metrics among other things. Measurement, therefore is both qualitative and quantitative. Metrics resides within measurement, but measurement also includes the qualifiable (tall, red, married, etc.) – values that do not have an associative number value, like metrics do.
Really Nice especially Good Examples super
Thanks Mam
in my opinion; indians fit to do the back-house-keeping job, because any client is not a native, we all work for the testing part of it, and all these metrics or so what ever! is just to swipe out interview process, or to protect their seat!, further once working for a PMO client, one of the super manager asked why some one do these activities, when its already planned by the PMO; all job is to regress test or re-test or capture defect in the client specified tool or xls ; report the status of defects, well testing never evolves because actual QA lies in the core industries and s/w is just naive or nothing!
It would be GREAT if u attach the excel sheet also…where we maintain the data(reports)
Hi Anuradha/Vijay thank you so much for this post. Can you please also share how to import the status directly from the HP QC, in the earlier tutorial of HP QC you have shared this but can you please dedicate a special write up for this test metrics and measurement in detail. At the end of the day the data’s and the presentation is the one which is going to shout loud behalf of us……:)
For those who want to have an excel template of this presentation to be able just to apply your numbers and get the results you are welcome to send me an email asking for it.
Would you please share the template you’re mentioning!
Hi Jordan, if you could provide a copy of the template that would great! Marybe.garth@gmail.com
Thanks, again!
Please share me excel, thank you!
I’ve started to learn testing ..!do have a doubt where the failed test cases are 26 and blocked is 9. then how the defects can be 30?
Test metrics related to number of scripts automated, number of scripts passing/failing can be included.
Hello everyone . I was working as QA TL, having experience of around two and half year. Right now I m in Bangalore . Any one can suggest me company name providing work from home for manual testing
Thanx
you can apply on crossover
Hi! I like this article, may i ask the authenticity and original reference to figures used in this article? e.g. metric life cycle and especially one that show measure and metrics relationship in hierarchical tree under “What Is Software Test Measurement?” heading. Thanks for your time and efforts to compile this information.
Nice explanation.
Can you help me? How can we map the software test metrics to software quality? Based on ISO 9126, the good characteristics of software product are functionality, usability, efficiency, reliability, maintainability and portability. Which software product quality characteristic that software test metrics measure?
Hi Jordan,
Please send the excel template
Thanks
Kanch
HAHAHAHAHHAHAHAHAHAHAHAHHAAH Correct bro broo brooo brah
Good article to understand the test metrices. Can you please provide industry standard for these metrics? It will definitely help.
Why the blocked test counted in Execution bucket, Number of tests executed should be 56 [ Test passed + test Failed], Blocked should be considered as Un executed
hi..thanks for your feedback…here in this article we have included the basic metrics used regularly in the projects…i will write detail post for project cost related metrics and metrics used in automation testing
Nice explanation on types of metrics.
One can refer puretestingbits.blogspot.com or testingpanda.com for more testing metrics such as load factor, size variance, Quality of fixes, % Risk mitigation efficiency, Review Effficiency etc.
Hi All
i am searching for a job for test lead.please tell which metrics are prepared by test lead.
It is very usefull and easily understand with example
Great article! thanks for your help! This site is giving so much quality testing info for free….long live this site.
Metrics is some thing you derive from Measurement.
It is like you measure for a period of time or multiple items and from the number you have collected you derive Metrics.
Eg. People in a house are of Different Age 15, 20, 25,30, 35
this is measurement, when you take an average of Age which will be 25 yrs(I.e (15+20+25+30+35)/5 =25), this 25 is Metrics.
Hope my answer gives you a better understanding
test execution percentage example
defect density example
we also use the metrics related to project cost
Just a simple question, is it true for DRE, the higher the better ?
Very good article
Hi,
How the estimates can be prepared before test execution starts.
What are the points to be covered in estimation except the resource,environment.
Calculate Defect Leakage in the Design where there are:
-15 Requirements
-15 Review Defects
-15 Design Review Defects
-20 Code Review Defects which includes 10 Design related defect captured during the Code Review
-20 System test Defect
Correction to % Test case Blocked metric:
Denominator should be Total no. of Test cases written or in scope. Pass and Fail only will account for Executed. If the test case is blocked, obviously, it cannot be executed. Hence, it has to be part of un-executed.
%ge Test cases Blocked: This metric is used to obtain the blocked %ge of the executed test cases. A detailed report can be submitted by specifying the actual reason of blocking the test cases.
%ge Test cases Blocked = (No. of Test cases Blocked / Total no. of Test cases written) * 100.
So, from the above data,
%ge Test cases Blocked = (9 / 100) * 100 = 9%
hi,,,,,,,,,,,,,,,,,, i want learn testing course can you please any one will guide me. With Regards
How can we really use this metrics in order to improve the quality of testing? What actions we can take in order to reduce the defect density