Test Coverage – What You Should Really Test?

Scene 8 years ago: when I was having 2 years of experience in software testing industry I was thinking that it was alright if I don’t know some fundamentals about software testing as something one can learn with experience and me too will do.

I was right – and wrong as well.

Right because with experience, you learn to see bigger picture, you understand real meaning of “Critical Situation” and you understand the end user more. Wrong because none of the mentioned things require experience, but early learning, which I understood very late. That is the encouraging factor to write this post.

Here is my experience from one of the interviews at that time:

How do you make sure that test coverage is complete or maximum? I was asked.

Ummmm……I make sure that I test every functionality, I said.

So you are saying that once you test all the functionalities, you feel that you have covered maximum of application/product, in terms of testing, the interviewer backfired.

Ummm…well….ummm….yes, because when you test all the functionalities, you are confident about application/product’s behaviour, I supported my answer.

I agree, but my question is – will it give you confidence that your test coverage is maximum or complete? the interviewer was in no mood to let me go.

Now, I was growing impatient, unknown about the fact that I was going to learn one of the most important topic about software testing – Test coverage”.

Rather than reproducing the excerpts of the interview, I am summarizing here, what I learned about Testing Coverage, on that day. But before that let’s be clear on one point – test coverage does not ever mean to know whether you are testing enough or not, neither it means you are testing perfectly or not. Having said that, presenting my understanding about test coverage.

Also read => How to improve code coverage.

Test coverage can have different meaning in different context. Let’s discover those contexts one-by-one:

What You Will Learn:

Product coverage – What aspects of product did you look at?

Yes, when test coverage is being measured in terms of product, the main area to focus on is – which areas of product you have tested and which remains untested?

Example #1: If “knife” is a product, you are testing; just do not concentrate on checking whether it cuts the vegetables/fruits properly. There are other aspects to look for such as – user should be able to handle it comfortably.

Example #2: If “notepad” is an application, you are testing, checking relevant features is a must thing. But other aspects to be taken care are – application responds properly while using other applications simultaneously, application does not crash when user tries to do something unusual, user is provided proper warning/error messages, user is able to understand and use the application easily, help content is available when required.

If you don’t look into the mentioned scenarios, you can’t claim that the test coverage for the application is complete.

Risk coverage – What risks have you tested for?

Risk coverage is an another aspect to have complete test coverage. You can’t tag the product or application as “tested”, until you test the associated risks too. Coverage of associated risks is an important factor of overall test coverage.

Example #1: While testing an airplane, if a tester tells you that he/she has fully tested the internal system of airplane and it’s working fine but only flying capability of airplane was not covered while testing – what would be your reaction? Well, that is what risk coverage means. Identifying risk as per the application/product and testing it thoroughly is always a good practice.

Example #2: While testing an e-commerce site, tester considered every effective factor but did not consider the risk of numbers of users accessing the website simultaneously and on the Super OFFER day, the not considered risk proved to be a huge mistake.

Also read => Risk based testing – How to Manage Risks During Test Planning Phase and risk management at test execution phase.

Requirements coverage – What requirements have you tested for?

If a product or application is developed very well but if it’s not matching with customer’s requirements then it’s of no use. Requirement coverage while testing is as important as product coverage.

Example #1: Excited for the family function, you asked tailor to stitch your dress and put on those peacock blue show buttons on the neckline. While stitching the dress, tailor thought that putting those buttons on neckline will not look good so he stitched a golden colored border instead. On the trial day, definitely, the unhappy customer shouted at tailor for not sticking to the requirements.

Example #2 : While testing a chat application, tester took care of all the important points like multiple users chatting in a group, two users chatting independently, all types of emoticons available, updates sent to user immediately etc. but forgot to look into requirement document, which clearly mentioned that when two users chat independently, video call option should be enabled. The client marketed the chat application claiming that it would allow calling, while two users chat independently. You can imagine what would have happen to the chat application.

Read also => How to Create Requirements Traceability Matrix

In conclusion

Test coverage does not end with the mentioned contexts. There are many other points that should be considered when it comes to test coverage but let’s keep that discussion for another day.

About author: This article is written by STH team member Bhumika M. She is a project lead, carrying 10+ years of software testing experience.

Pour in your comments and views about the post as well as test coverage. As usual, we love to hear from you.