Important Software Test Metrics and Measurements – Explained with Examples and Graphs

In software projects, it is most important to measure the quality, cost and effectiveness of the project and the processes. Without measuring these, project can’t be completed successfully.

In today’s article we will learn with examples and graphsSoftware test metrics and measurements and how to use these in software testing process.

There is a famous statement: “We can’t control things which we can’t measure”.

Here controlling the projects means, how a project manager/lead can identify the deviations from the test plan ASAP in order to react in the perfect time. Generation of test metrics based on the project needs is very much important to achieve the quality of the software being tested.

Software Testing Metrics and measurements

What are Software Testing Metrics?

A Metric is a quantitative measure of the degree to which a system, system component, or process possesses a given attribute. 

Metrics can be defined as “STANDARDS OF MEASUREMENT”.

Software Metrics are used to measure the quality of the project. Simply, Metric is a unit used for describing an attribute. Metric is a scale for measurement.

Suppose, in general, “Kilogram” is a metric for measuring the attribute “Weight”. Similarly, in software, “How many issues are found in thousand lines of code?”, here No. of issues is one measurement & No. of lines of code is another measurement. Metric is defined from these two measurements.

Test metrics example:

  • How many defects are existed within the module?
  • How many test cases are executed per person?
  • What is the Test coverage %?

What is Software Test Measurement?

Measurement is the quantitative indication of extent, amount, dimension, capacity, or size of some attribute of a product or process.

Test measurement example: Total number of defects.

Please refer below diagram for clear understanding of the difference between Measurement & Metrics.

difference between Measurement & Metrics

Why Test Metrics?

Generation of Software Test Metrics is the most important responsibility of the Software Test Lead/Manager.

Test Metrics are used to,

  1. Take the decision for next phase of activities such as, estimate the cost & schedule of future projects.
  2. Understand the kind of improvement required to success the project
  3. Take decision on process or technology to be modified etc.

Importance of Software Testing Metrics:

As explained above, Test Metrics are the most important to measure the quality of the software.

Now, how can we measure the quality of the software by using Metrics?

Suppose, if a project does not have any metrics, then how the quality of the work done by a Test analyst will be measured?

For Example: A Test Analyst has to,

  1. Design the test cases for 5 requirements
  2. Execute the designed test cases
  3. Log the defects & need to fail the related test cases
  4. After the defect is resolved, need to re-test the defect & re-execute the corresponding failed test case.

In above scenario, if metrics are not followed, then the work completed by the test analyst will be subjective i.e. the test report will not have the proper information to know the status of his work/project.

If Metrics are involved in the project, then the exact status of his/her work with proper numbers/data can be published.

I.e. in the Test report, we can publish:
1. How many test cases have been designed per requirement?
2. How many test cases are yet to design?
3. How many test cases are executed?
4. How many test cases are passed/failed/blocked?
5. How many test cases are not yet executed?
6. How many defects are identified & what is the severity of those defects?
7. How many test cases are failed due to one particular defect? etc.

Based on the project needs we can have more metrics than above mentioned list, to know the status of the project in detail.

Based on the above metrics, test lead/manager will get the understanding of the below mentioned key points.
a) %ge of work completed
b) %ge of work yet to be completed
c) Time to complete the remaining work
d) Whether the project is going as per the schedule or lagging? etc.

Based on the metrics, if the project is not going to complete as per the schedule, then the manager will raise the alarm to the client and other stake holders by providing the reasons for lagging to avoid the last minute surprises.

Metrics Life Cycle:

Metrics Life Cycle

Types of Manual Test Metrics:

Testing Metrics are mainly divided into 2 categories.

  1. Base Metrics
  2. Calculated Metrics

Base Metrics:
Base Metrics are the Metrics which are derived from the data gathered by the Test Analyst during the test case development and execution.

This data will be tracked throughout the Test Life cycle. I.e. collecting the data like, Total no. of test cases developed for a project (or) no. of test cases need to be executed (or) no. of test cases passed/failed/blocked etc.

Calculated Metrics:
Calculated Metrics are derived from the data gathered in Base Metrics. These Metrics are generally tracked by the test lead/manager for Test Reporting purpose.

Examples of Software Testing Metrics:

Let’s take an example to calculate various test metrics used in software test reports:

Below is the table format for the data retrieved from the test analyst who is actually involved in testing:

Example of Software Testing Metrics

------------

Definitions and Formulas for Calculating Metrics:

#1) %ge Test cases Executed: This metric is used to obtain the execution status of the test cases in terms of %ge.

%ge Test cases Executed = (No. of Test cases executed / Total no. of Test cases written) * 100.

So, from the above data,
%ge Test cases Executed = (65 / 100) * 100 = 65%

#2) %ge Test cases not executed: This metric is used to obtain the pending execution status of the test cases in terms of %ge.

%ge Test cases not executed = (No. of Test cases not executed / Total no. of Test cases written) * 100.

So, from the above data,
%ge Test cases Blocked = (35 / 100) * 100 = 35%

Test metrics graph 1

Test metrics graph 2

#3) %ge Test cases Passed: This metric is used to obtain the Pass %ge of the executed test cases.

%ge Test cases Passed = (No. of Test cases Passed / Total no. of Test cases Executed) * 100.

So, from the above data,
%ge Test cases Passed = (30 / 65) * 100 = 46%

#4) %ge Test cases Failed: This metric is used to obtain the Fail %ge of the executed test cases.

%ge Test cases Failed = (No. of Test cases Failed / Total no. of Test cases Executed) * 100.

So, from the above data,
%ge Test cases Passed = (26 / 65) * 100 = 40%

#5) %ge Test cases Blocked: This metric is used to obtain the blocked %ge of the executed test cases. A detailed report can be submitted by specifying the actual reason of blocking the test cases.

%ge Test cases Blocked = (No. of Test cases Blocked / Total no. of Test cases Executed) * 100.

So, from the above data,
%ge Test cases Blocked = (9 / 65) * 100 = 14%

Test metrics graph 3

Test metrics graph 4

#6) Defect Density = No. of Defects identified / size

(Here “Size” is considered as requirement. Hence here the Defect Density is calculated as number of defects identified per requirement. Similarly, Defect Density can be calculated as number of Defects identified per 100 lines of code [OR] No. of defects identified per module etc.)

So, from the above data,
Defect Density = (30 / 5) = 6

#7) Defect Removal Efficiency (DRE) = (No. of Defects found during QA testing / (No. of Defects found during QA testing +No. of Defects found by End user)) * 100

DRE is used to identify the test effectiveness of the system.
Suppose, During Development & QA testing, we have identified 100 defects.
After the QA testing, during Alpha & Beta testing, end user / client identified 40 defects, which could have been identified during QA testing phase.

Now, The DRE will be calculated as,
DRE = [100 / (100 + 40)] * 100 = [100 /140] * 100 = 71%

$8) Defect Leakage: Defect Leakage is the Metric which is used to identify the efficiency of the QA testing i.e., how many defects are missed / slipped during the QA testing.

Defect Leakage = (No. of Defects found in UAT / No. of Defects found in QA testing.) * 100

Suppose, During Development & QA testing, we have identified 100 defects.
After the QA testing, during Alpha & Beta testing, end user / client identified 40 defects, which could have been identified during QA testing phase.

Defect Leakage = (40 /100) * 100 = 40%

#9) Defects by Priority: This metric is used to identify the no. of defects identified based on the Severity / Priority of the defect which is used to decide the quality of the software.

%ge Critical Defects = No. of Critical Defects identified / Total no. of Defects identified * 100
From the data available in the above table,
%ge Critical Defects = 6/ 30 * 100 = 20%

%ge High Defects = No. of High Defects identified / Total no. of Defects identified * 100
From the data available in the above table,
%ge High Defects = 10/ 30 * 100 = 33.33%

%ge Medium Defects = No. of Medium Defects identified / Total no. of Defects identified * 100
From the data available in the above table,
%ge Medium Defects = 6/ 30 * 100 = 20%

%ge Low Defects = No. of Low Defects identified / Total no. of Defects identified * 100
From the data available in the above table,
%ge Low Defects = 8/ 30 * 100 = 27%

Test metrics graph 5

Recommended reading => How to Write an Effective Test Summary Report

Conclusion:

The metrics provided in this article are majorly used for generating the daily/weekly status report with accurate data during test case development/execution phase & this is also useful for tracking the project status & Quality of the software.

About the author: This is a guest post by Anuradha K. She is having 7+ years of software testing experience and currently working as a consultant for a MNC. She is also having good knowledge of mobile automation testing.

Which other test metrics do you use in your project? As usual, let us know your thoughts/queries in comments below.

Recommended reading

39 comments ↓

#1 Irina Ivanova

In my opinion, difference between metrics and measurements is described quite messy in the article.

“Suppose, in general, “Kilogram” is a metric for measuring the attribute “Weight””
–Kilogram is rather measurement, than metric (which is also shown in Measurement & Metrics diagram, where Km and Hours are measurements).

All 13 testing metrics in example table are rather measurements.

#2 Nirav

we also use the metrics related to project cost

#3 Paresh Mahajan

Test metrics related to number of scripts automated, number of scripts passing/failing can be included.

#4 Karuvarasan.s

Hi Anuradha/Vijay thank you so much for this post. Can you please also share how to import the status directly from the HP QC, in the earlier tutorial of HP QC you have shared this but can you please dedicate a special write up for this test metrics and measurement in detail. At the end of the day the data’s and the presentation is the one which is going to shout loud behalf of us……:)

#5 rajeshri

Hello everyone . I was working as QA TL, having experience of around two and half year. Right now I m in Bangalore . Any one can suggest me company name providing work from home for manual testing
Thanx

#6 G K Ankem

hi,,,,,,,,,,,,,,,,,, i want learn testing course can you please any one will guide me. With Regards

#7 Anuradha Kanamarlapudi

hi..thanks for your feedback…here in this article we have included the basic metrics used regularly in the projects…i will write detail post for project cost related metrics and metrics used in automation testing

#8 Nick

Hi,

It’s something i really want to learn. But at present i am leading a project where test cases creation and execution were not used.

So please suggest me how to create test metrics based on number issue found and number of issues closed/reopened.

Thanks

#9 yaser

Really Nice especially Good Examples super

Thanks Mam

#10 Dipankar

Just a simple question, is it true for DRE, the higher the better ?

#11 Arvind

Just came across this post, pertaining to comments..
@rajeshri : i have been working for UHG in bangalore, Its a brilliant company to work(Fortune14). They provide you with plenty of WFH. I have been working as QA TL, I used to work from my home(Chennai) 2weeks everymonth.

#12 Arvind

@Dipankar: It is obvious. DRE should be higher, it should be nearly 100%. If it is 100% then it represents testing is carried our perfectly hence there’s no UAT defects/Customer issues.

DRE should be higher and Defect Density should be lower(viz good for entire project)

#13 VENkAT

It would be GREAT if u attach the excel sheet also…where we maintain the data(reports)

#14 HARI

@ All

Does any one have a sample excel sheet where we show all the defect details and test case reports in the call or to the client in the form of Graph or Pie chart( with formula based in excel sheet).If any please share

#15 Arnav

What is the difference between test metric & traceability matrix?

#16 ameer

Its really good material, if you have few more metrics then its excellent like test coverage, defect aging

#17 Prashant Patil

Thanks for your post.
There are more metrics used in our project. I am listing few of them,

Requirement/Usecase Coverage
Test case/Script design productivity
Test case/Script execution productivity
Test Effectiveness
Cost Of Quality (COQ)
Effort Variance
(Overall and Variance at each milestone)
Schedule variance

#18 Inder

Great article! thanks for your help! This site is giving so much quality testing info for free….long live this site.

#19 Jordan

For those who want to have an excel template of this presentation to be able just to apply your numbers and get the results you are welcome to send me an email asking for it.

#20 Very good article

Very good article

#21 pradeep

how to give severity and priority for the below equations
1+0=0
1*0=1
4+5=9
4*1=41

#22 hameetha

How can we really use this metrics in order to improve the quality of testing? What actions we can take in order to reduce the defect density

#23 siva

in my opinion; indians fit to do the back-house-keeping job, because any client is not a native, we all work for the testing part of it, and all these metrics or so what ever! is just to swipe out interview process, or to protect their seat!, further once working for a PMO client, one of the super manager asked why some one do these activities, when its already planned by the PMO; all job is to regress test or re-test or capture defect in the client specified tool or xls ; report the status of defects, well testing never evolves because actual QA lies in the core industries and s/w is just naive or nothing!

#24 Connor Roberts

The difference between metrics and measurement is simply. Metrics are mathematical (quantifiable), and can have a number applied to them. Measurement is the larger body, that contains metrics among other things. Measurement, therefore is both qualitative and quantitative. Metrics resides within measurement, but measurement also includes the qualifiable (tall, red, married, etc.) – values that do not have an associative number value, like metrics do.

#25 siva

Hi,
How the estimates can be prepared before test execution starts.
What are the points to be covered in estimation except the resource,environment.

#26 Nageshwar

Nice explanation on types of metrics.

One can refer puretestingbits.blogspot.com or testingpanda.com for more testing metrics such as load factor, size variance, Quality of fixes, % Risk mitigation efficiency, Review Effficiency etc.

#27 Saleem Javed

If I consider the re-execution of the test cases due to failure rate, how does the calculations go for these metrics, especially when you are calculating the

– %age of Test Cases Failed or Passed

#28 Zu

Nice explanation.

Can you help me? How can we map the software test metrics to software quality? Based on ISO 9126, the good characteristics of software product are functionality, usability, efficiency, reliability, maintainability and portability. Which software product quality characteristic that software test metrics measure?

#29 Manoj

Why the blocked test counted in Execution bucket, Number of tests executed should be 56 [ Test passed + test Failed], Blocked should be considered as Un executed

#30 Paro Rani

HAHAHAHAHHAHAHAHAHAHAHAHHAAH Correct bro broo brooo brah

#31 I am AJ

Metrics is some thing you derive from Measurement.
It is like you measure for a period of time or multiple items and from the number you have collected you derive Metrics.
Eg. People in a house are of Different Age 15, 20, 25,30, 35
this is measurement, when you take an average of Age which will be 25 yrs(I.e (15+20+25+30+35)/5 =25), this 25 is Metrics.
Hope my answer gives you a better understanding

#32 Ganesan

It is very usefull and easily understand with example

#33 AB

If i have executed 100% cases and all my test cases Pass, do we deliver and tell customer that my pass percent is 100%

#34 Udi

Would you please share the template you’re mentioning!

#35 harish

Hi Jordan,

Please send excel template

#36 Rajendra ohdar

Easy understand nice good example
Thankyou

#37 Kiran

Calculate Defect Leakage in the Design where there are:
-15 Requirements
-15 Review Defects
-15 Design Review Defects
-20 Code Review Defects which includes 10 Design related defect captured during the Code Review
-20 System test Defect

#38 @pradip

1+0=0 is P2
1*0=1 is P1
4+5=9 is P4
4*1=41 is P3

#39 Neve

Please share me excel, thank you!

Leave a Comment