Overview of Acceptance Test Report (Part-III):
In our previous tutorial on “Acceptance Testing Documentation with Real-Time Scenarios” we discussed Acceptance Test plan.
In this tutorial, we will take an in-depth look at reporting of Acceptance Test Status, Acceptance Test Summary, and Sign-Off.
Some generic templates are included in this tutorial to enhance your understanding in a better way. We will also hover over the concept of Acceptance Testing in Agile and Acceptance Test Driven Development.
In short, this tutorial will explain you about Acceptance test Status Report and Summary report along with some generic templates for your clear understanding and also brushes up the concept of Acceptance testing in Agile and test-driven development in an easily understandable way.
What You Will Learn:
- Acceptance Test Status Report
- Acceptance Test Summary Report
- Acceptance Testing in Agile
- Who does Acceptance Testing in Agile?
- Benefits of Acceptance Testing in Agile
- Acceptance Test Driven Development (ATDD)
- Recommended Reading
Acceptance Test Status Report
Acceptance Test Report should always summarize the acceptance tests that are performed along with their results. It should be addressed to all the identified stakeholders who are a part of Acceptance Testing Phase. Once the execution of Acceptance Tests have started, the progress should be reported on a day-to-day basis.
Generic Template for Acceptance Test Status Report:
Date: Acceptance Test Status Report Date
Today’s Acceptance Tests Execution details:
- Number of Tests Passed
- Number of Tests Failed
- Number of Tests In-Progress
Acceptance Tests Execution Details till date:
- Total number of Tests
- Number of Tests Passed
- Number of Tests Failed
- Number of Tests In-Progress
- Number of Tests Pending
- Number of Defects Logged
- Each Defect should have the below details:
- ID, Summary, Component, Severity
- The total number of defects logged till now (in Acceptance Testing Phase).
This report has to be reviewed on a daily basis to ensure that the execution is on track and there is no deviation from the planned schedules.
Acceptance Test Summary Report
This is the report which summarizes the status of the entire Acceptance Testing phase. This involves details like testing activities conducted, references to criteria met, requirement specifications, business rules, execution results, planned schedules, deviations, etc.
Generic Template for Acceptance Test Summary Report:
<This is to summarize about acceptance testing activities that took place i.e., Acceptance tests design, Acceptance Test Execution, Defects, Test environment. It also includes details like Release of the product, reference to Acceptance Test Plan, Entry Criteria passed, Exit criteria to be met, requirements/business requirement specifications, and business rules.>
<This helps to improve the Acceptance Test planning in the future releases.>
<Any reason for tests not executed should be addressed immediately. This could be due to dependencies, environment issues, etc. Also, the Test plan should be reviewed for addressing the cases for these kinds of issues).>
<Evaluation should be done for each component under test, by analyzing the success rate based on test execution and defects logged. Also, Entry criteria should be evaluated for actually met. Similarly, exit criteria should be confirmed based on the Acceptance test results of the execution. Deviations should be addressed with correct reasons and possible action items to correct it in the future releases.>
<Based on the entire report, recommendation for the product either to be released to the market or reject is provided. Any improvements required, suggestions from acceptance testers, defects severity, and Pass rate impacts the recommendation for the product to be launched in the market.>
<Actual effort spent for each of the activities in the phase should be explained in detail.>
Once the Product has passed Acceptance Testing, it will be recommended to Go Live. Before it is launched to Production, it has to be Signed-Off formally.
Generic Template for Sign-Off Report:
Product Name, Release Version, Build Number
<Mention Product Name and its Release Version. Mention the latest build number that underwent Acceptance Testing.>
<Attach the latest Acceptance Test Report.>
<Mention the date when the latest Acceptance Test Report was reviewed.>
<Mention by whom the latest Acceptance Test Report was reviewed.>
<Mention all the review comments on Acceptance Test Report.>
<Mention the date when the Acceptance testing is Signed-off.>
<Mention who Signed-off the Acceptance Testing for the product.>
<Provide the statement for Go decision for the Product.>
In general, any of the above reports should be reviewed by major stakeholders for its template and has to be agreed upon what has to go as the information within.
All the details that get filled into the report should be cross-checked before sharing it with the Stakeholders. Any discrepancies in the report will highly impact the Business decision and could result in Product Failure in the Market.
Hence, reporting should always be handled by specialists or senior members of the team.
Acceptance Testing in Agile
In Agile, Acceptance Criteria of each User story is targeted for Acceptance Tests, i.e., Acceptance tests are derived from the Acceptance criteria of a user story. Each Acceptance Criteria can have one or more Acceptance Tests to cover the scenario.
Acceptance Tests are usually designed by a QA who is the Subject Matter Expertise in the area. Acceptance Testing in Agile starts much early when compared to the other approaches, usually within the sprints itself.
It is performed very frequently as each sprint will have new User stories coming up and also enhancements/continuation of the previous stories.
Acceptance Testing is performed at two different stages in Agile:
- When the feature is created and in its initial stage – basic.
- When the feature is integrated and stabilized with the other features of the product.
Each user story here has to undergo Acceptance Test and should be passed for consideration. Any failures in Acceptance test should be considered as a high priority and fixed immediately, this, in turn, will have the acceptance test to execute it.
Story points are given to each user story based on the success of Acceptance test results for each of the acceptance criteria. Acceptance Testing also defines the completion at User Story level, stating that the Acceptance criteria for the story are met.
Who does Acceptance Testing in Agile?
Usually, Product managers, Subject Matter Expertise (could be Customer AND/OR Beta testers) perform Acceptance Testing in an Agile environment. Sometimes, QA also involves in this activity along with their regular regression tasks.
Benefits of Acceptance Testing in Agile
There are several benefits of Acceptance Testing in Agile.
- Closer collaboration between Product manager and the team.
- Builds confidence at the User Story level.
- Will help to derive more scenarios to cover each Acceptance criteria.
- Increased probability to improvise the solutions to the product through Acceptance criteria in User Stories.
Though there are several benefits, there are certain demerits too.
- Not all the stories can be considered for Acceptance testing. Only functional stories to be covered – Story-wise coverage may come down.
- Not all the Acceptance Criteria can be considered for Acceptance testing. Only functional criteria are to be covered – Acceptance Criteria wise coverage within User story may come down.
- Since stakeholders from different backgrounds are involved and as story-wise acceptance testing is performed directly, it is quite difficult for everyone to be on the same page (basically in understanding the level at individual User story).
- Since the release duration is less when compared to other approaches, it is quite difficult to accommodate Acceptance Testing within Sprints.
Acceptance Test Driven Development (ATDD)
This is one of the Agile development practice where the entire team collaboratively discusses each of the Acceptance criteria of User Story and builds strong Acceptance Tests around them.
This is because different perspectives from each of the team member will give a new way of thinking for each of the Acceptance criteria and will arrive with a good number of acceptance tests covering more scenarios. Sometimes, ATDD is also called Story Test Driven Development (STDD).
Actually, ATDD happens before the development starts. So, the developers, in this approach, will know what actually is expected and how to achieve it. The entire team will share a common understanding of the feature and what is being built.
This describes how the product is being built and in turn, will give a fair idea of how the product will actually function before it is handed over for testing. Hence it is termed as “Acceptance Test Driven Development”.
Acceptance Testing in any of its approach has the common goal of building Customer confidence and Satisfaction on the Product that is developed before it goes Live. This is gained only when there are no/fewer low severity defects in the product that do not hamper any of the functionalities.
In a nutshell:
- Acceptance Tests are Passed.
- Defects are in Acceptable levels.
- Flow-wise/Scenario-wise coverage achieved.
- Product and its solutions are accepted.
- The customer is confident enough in the product.
- All the Product documents are updated to match latest functions.
- Result for team’s effort.
- Good to go forward with the Production Launch.
Hope you would have gained immense knowledge from these Acceptance Testing Tutorials. Feel free to share your thoughts and put forward your questions in the comments section below.