This is the concluding part of our “Software Testing training on a live project” series.
It is going to be about defects and also a few remaining topics that will mark the completion of the Test Execution phase of the STLC.
In the previous article, while Test Execution was going on, we encountered a situation where the expected result of the test case was not met. Also, we identified some unexpected behavior during Exploratory Testing.
What happens when we encounter these deviations?
We obviously have to record them and track them to make sure that these deviations get handled and eventually fixed on the AUT.
#1) These deviations are referred to as Defects/bugs/issues/incidents/errors/faults.
#2) All the following cases can be logged as defects
- Missing requirements
- Incorrectly working requirements
- Extra requirements
- Reference document inconsistencies
- Environment-related issues
- Enhancement suggestions
#3) Defect recording is mostly done in excel sheets or via the use of a Defect Management software/tool. For information on how to handle defects via tools, try using the following links:
- HP ALM
- Atlassian JIRA
- Also, refer to this post for a list of the most popular Bug Tracking tools in the market.
What You Will Learn:
How To Log The Defects Effectively
We will now try to see how to log the defects we encountered in the previous article in an excel sheet. As always, choosing a standard format or template is important.
Typically, the following columns are a part of the Defect Report:
- Defect ID: For unique identification.
- Defect Description: This is like a title to describe the issue briefly.
- Module/section of the AUT: This is optional, just to add more clarity as to indicate the area of the AUT where the problem was encountered.
- Steps to Reproduce: What is the exact sequence of operations to be performed on the AUT to recreate the bug are to be listed here. Also, if any input data is specific to the problem that information is to be entered as well.
- Severity: To indicate the intensity of the issue and eventually the impact this might have on the functioning of the AUT. The guidelines on how to assign and what values to assign in this field can be found in the test plan document. So, please refer to the Test Plan document from article 3.
- Status: Will be discussed further in the article.
- Screenshot: A snapshot of the application to show the error when it happened.
These are some of the ‘must-have’ fields. This template can be expanded (E.g. to include the name of the tester who reported the issue) or contracted (For Example, the module name removed) as needed.
Following the above guidelines and using the template above, a sample Defect log/Report could look like this:
Sample Defect Report for OrangeHRM Live project:
=> Click here to download live project Defect Report
Below is the sample Defect Report created in qTest Test Management tool: (Click on image to enlarge)
Defects are no good when we log them and keep them to ourselves. We will have to assign them in the right order to have the concerned teams act on them. The process – who to assign or what order to follow can also be found in the test plan document. It is mostly similar to (Click on image to enlarge)
From the above process, it can be noted that bugs go through different people and different decisions in the process of being identified to fixed. To track and to establish transparency as to exactly what state a certain bug is at, a “Status” field is used in the bug report. The entire process is referred to as a “Bug life cycle”. For more information on all the statuses and their meanings, please refer to this Bug Life Cycle tutorial.
A Few Pointers While Bug Tracking
- When we are new to a creative Team/Project/AUT, it is always best to discuss the issue we encountered with a peer to make sure that our understanding of what really makes for a defect is correct or not.
- To provide all the information that is necessary to reproduce the issue. A defect that comes back to a testing team with the status set as “Not enough information” does not reflect very positively on us. Check out this post – How to get your all bugs resolved without any ‘Invalid bug’ label.
- Check if a similar issue was raised before creating a new one. ‘Duplicate’ issues are also bad news for the QA team.
- If there is an issue, that comes up randomly and we don’t know the exact steps/situations in which we can recreate the issue- raise the issue all the same. At the risk of the issue being set to “IrReproducible/not enough information” – we still have to make sure that we handled all possible malfunctions to the best extent possible.
- The general practice is that the QA team creates each one’s defects in an excel sheet during a day and consolidates it at the end of the day.
The Complete Defect Life Cycle
For our live project if we were to follow the defect life cycle for defect 1,
- When I (the tester) create it, its status is “New”. When I assign it to the QA team lead, the status is still “New” but the owner is now the QA lead.
- The QA lead will review the issue and on determining that it is a valid issue, the issue is assigned to the Dev lead. At this phase, the status is “Assigned” and the owner is Dev lead.
- The dev lead will then assign this issue to a developer who will work on fixing this issue. The status will now be “Work in Progress” (or something similar to that effect), the owner is the developer.
- For defect 1, the developer is not able to reproduce the error, so he assigns it back to the QA team and set the status as “Not able to reproduce”.
- Alternately, if the developer was able to work on it and fix an issue, the status would be set to “resolved” and the issue would be assigned back to the QA team.
- QA team will then pick it up, retest the issue and if it is fixed, will set the status to “Closed”. If the issue still exists, the status is set to “Reopen” and the process continues.
- Depending on the other situations, the status can be set as “Deferred”, “Not enough information”, “Duplicate”, “working as intended”, etc by the developer.
- This method of recording the defects, reporting and assigning them, managing them is one of the major activities performed by the QA team members during the test execution phase. This is done on a daily basis until a particular test cycle is complete.
- Once Cycle 1 is done, the dev team will take a day or two to consolidate all the fixes and rebuild the code into the next version that will be used for the next cycle.
- The same process again continues for cycle 2 as well. At the end of the cycle, there is a chance that there might still be some issues “Open” or unfixed in the application.
- At this stage- do we still continue with Cycle 3? If yes, when will we stop testing?
Exit Criteria for the OrangeHRM Live Project Testing
This is where we employ what we would call the “Exit Criteria”. This is pre-defined in the Test Plan document. It is simply in the form of the checklist that will determine whether we conclude the testing after cycle 2 or go for one more cycle. It looks like, the below when filled out taking into consideration some hypothetical answers to the following questions concerning, OrangeHRM project:
When we look carefully at the above checklist, there are metrics and sign off mentioned there that we have not discussed earlier. Let us talk about them now.
We have established that during the Test Execution phase, reports are sent out to all the other project team members to give a clear idea about what is happening in the QA Execution phase. This information is important to everyone in order to get validation about the overall quality of the final product.
Imagine I report that 10 test cases passed or 100 test cases were executed- these numbers are merely raw data and do not give a very good perspective about how things are going on.
Metrics play a vital role in filling this gap. Metrics are in simple words, intelligent numbers that the testing team collects and maintains. For Example, if I said 90% of the test cases passed, it makes more sense than saying 150 test cases passed. Isn’t it?
There are different kinds of Metrics collected during the test execution phase. What metrics exactly are to be collected and maintained for what periods of time- this information can be found in the test plan document.
The following are the most commonly collected test metrics for most projects:
- Pass Percentage of the Test cases
- Defects density
- Critical Defects percentage
- Defects, Severity wise number
Check out the Status Report attached to this article to see how these metrics are used.
Test Sign Off /Completion Report
As we have to notify all the stakeholders that testing has begun, it is also the QA team’s duty to let everyone know that testing has been completed and share the results. So, typically an email is sent from the QA team (usually the Team Lead/QA Manager) giving an indication that the QA team has signed off on the product attaching the test results and the list of open/known issues.
Sample Test Sign off Email:
To: Client, PM, Dev team, DB team, BA, QA team, Environment Team (and anyone else that needs to be included)
Email: Hello Team,
QA team signs off on the OrangeHRM version 3.0 software after the successful completion of the 2 cycles of functional testing the website.
The test cases and their execution results are attached to the email. (Or mention the location where they are present. Or if using test management software, provide details regarding the same.)
The list of known issues is attached to the email too. (Again, any other references that make sense can be added.)
QA team lead.
Attachments: Final Execution Report, Final issue /defect report, Known issues list
Once the Test sign off email goes out from the QA team, we are officially done with the STLC process. This does not necessarily mark the completion of the “Test” phase of the SDLC. We still have the UAT testing to finish for that to happen. Find more details about UAT testing here.
After the UAT is done, the SDLC moves to deploy phase where it goes live and is available to its customers/end-users to be consumed.
This has been our effort to bring the most live like QA Project experience to our readers. Please let us know your comments and questions about this free online Software Testing QA training series.
67 thoughts on “Live Project Bug Tracking, Test Metrics, and Test Sign off – Free QA Training Day 6”
This article is very usefull I learned so many things in this article.
Hi i am looking for a job change i have 2.7 years of exp in software testing please mail me any openings email@example.com
This was superb series. leant a lot from it. Keep us updated with similar articles.
Great Series….Hats off to the team
Very nice article… useful information of bug tracking.
Thanks for this free long days tutorials. I will like to reread this articles again. Is it possible if you include this tutorial with software career package eBook as update.
Always find your articles awesome. It will help beginners a lot.
There is one important section missing in the bug details: what is the expected behavior. Without this you only say “something is not working” without adding “how it should work”.
P.S. of course this does not apply in cases of application crashes where the expected behavior is clear
Great explanation. Thanks to STH team. This helps a lot me and maybe others.
very nice and useful articles………..
Very nice article for manual testers
Fantastic, Superb !!
Have no enough words to appreciate the STH team for the fabulous series . Very helpful to the testers( specially for who are new to this field).
thanks for the sample signoff email. It helps a lot.
Hope the same in future articles.
It is doubt clearing session with wonderful clarification.
Actually came to know about more aspects of real time testing projects through this article only! keep up the great work. cheers!
this is very informative and explained in easy language by live project.
Its very helpfull
Thank you so much…Its worthy…
Awesome job … Really informative ..worth reading ..
There is a issue/bug/defect in this blog … jus now I added a comment .. the time was 11:00 PM IST may be.. but it is showing 5:28 PM …. Is it the server time not sure ? LOL.. :)
really vlued article, thanks so much
Hi,This is very superb tutorial,it is very helpful to begginer’s.thanks to STH team to give such valuable thing for testers.
Q)What are the different tools for Agile?
A) Some of the important commercial tools available for agile project management: Jira GreenHopper, VersionOne, Rally, Team Foundation (by Microsoft), Mingle (by ThoughtWorks)
Thanks so much for sharing this series it is invaluable! The SRS documents was very helpful. I am new to QA and this really helped me to understand the fundamentals of the STLC. Keep up the great work!
Hello STH Team,
Really learnt a lot of new things related to Testing.
Thanks a ton !!!!!!!!!!!
Appreacite your efforts in writing this blog..
its very helpful
Thanks to STH team
Thank You STH team to putting your efforts and giving real time experience which is really helpful .
Can you please ex[lain about Jmeter.and Performance and load testing using it.How to analyse the results in jmeter.
Thanks for summarizing the whole qa process at once
Thank you for this tutorial. I found it very helpful and direct. Also, all the links you provided during the tutorial are great and very useful to understand all the STLC.
Click here to download live project Defect report. ….this link is not working could you please forward me to my mail-id or please make that link to work.it could help me lot.
Your live tutorial is very useful..!!!!!!!!!
Thanks for the tutorial!!
Just finished it! I hope I will pass my interview for QA tom! :D
can i know a test engineer how may test cases write per day?
Great effort to clear practical picture……..it is really very helpful for freshers……hats off to STH Team.
This was brilliant and very helpful! Great work guys. Thank you!
its nice to upgrade my QA skills
Wow…exceptional educational series.
Hats off !
Great Job !
Knowing just Definitions is not good enough..
The effort made was really too good.
This bought more confidence in me..Thank u !
superb explanation. Searched many websites to find this kind of stuff ………..finally i say it is softwaretestinghelp website which helped me
realy very helpfull………..plzzzz share some more topic where tester can become more knowledgable…………
ITS VERY HELPFULL TO BEGINERS….@
KUDOS! Really appreciate all the effort you took for all of this project.
Great work!I am benefited with your tutorials.
Awesome series guys!!!!!!!
It’ s really very helpful to clear the testing concept….
Extremely helpful…Appreciate all the efforts to put up such a comprehensive article
Very nicely explained, even u have covered the minute details. Thanks a ton for sharing to us. Learnt many things…thanks again
Great helping ! This is so useful and meaningful for me.
Thanks a lot, th Owner !
hats off this person who developed this series. I had been searching for complete testing project for several years. now this day I got complete picture of testing project. I do not know ,what I have to say about this author. I am seeking almighty to bless this author.
This online training 1-6 has been my best so far. Thank you very much for the simplicity and great examples. I am now confident to work with a project from beginning to end. Thumbs up to you all!
Well-done, the best intro to SDLC and STLC I’ve ever seen, mostly thanks to real-world like documents and hands-on approach
well done STH
Great helping i learnt lot
Thanks a lot! Great QA Training!
thanks for this awesome tutorials.It helps me a lot.well done STH team :)
Excellent tutorials thanks for the help.
Thank you so much for this tutorials. This give me the confident to make a career in testing. This is the best tutorial I ever read. Thanks a lot to STH team..
It was really good learning through this mini series tutorials. Thanks STH team.
hi sir i am using selenium for Automation can you tell me which oen source bug management tool i have to use
Hi, very nice tutorial. I am searching for the long time. I got certifications in automation testing ‘ certified software testing professional’. I got good grade also. But unfortunately for personal things I couldn’t not work for 8 years. But I have 8 years experience in business analyst role. Please suggest me a career path to get entry into testing. Can I get jobs in testing?
Thank you swati. You’re the best !
Thanx a LOT, this was great, GOD bless you all who did those things, wish you all the best in life!
Thank you for the tutorials. I hope you can continue to keep them available and updated for a long time to come.
Hat’s off to the designer of this end to end project. Many doubts got cleared in due to his/her endeavor. There is a real need of these kind of projects. Though it’s quite old it’s really informative!
Please design new projects with latest techniques with due consideration of industry’s agility.
Thanks A lot, This has been really informative and amazing.