Documentation of Acceptance Testing (Part- II):
This tutorial is the continuation of our previous tutorial where we discussed what is Acceptance testing, when it has to be done, who does it, its importance, types, process, impact on different teams, etc.
Documents play a very important role in Acceptance Testing and any issues with regards to the document have a huge negative impact. When a proper check is not exercised, then it may even lead to the Failure of the Product.
In this tutorial, we will learn more about the different documentation involved in Acceptance Testing i.e., Acceptance Test Plan, Test Plan Review Checklist, Acceptance Test Template, examples based on real-time scenarios, how to identify and write acceptance tests, etc in detail.
What You Will Learn:
Acceptance Test Plan
Like any other test plan, Acceptance test plan also includes some components like Scope, Approach, Test Environment, Resources, Responsibilities, Acceptance Tests references, Entry criteria, Exit criteria, Tools, etc.
The only thing that differentiates Acceptance test plan from a regular test plan is its factors that result in Business Decision. Acceptance Test Plan is one of the vital documentation that provides guidance on how to perform acceptance testing for a particular project.
Acceptance test plan has to be reviewed and approved before Acceptance Test Execution. All the subsequent changes again have to undergo review & approval process and has to be in-track.
Acceptance Test Plan review is usually done by Managers/Business Analysts/Customers.
Key points to be considered while designing Acceptance Test Plan:
- It should be Detailed and Specific. Must include only what is required for testing and what information is necessary for the team to carry out testing.
- It should be Clear and Concise. No ambiguity. If at all there is something which may lead to confusion, then elaborate it, but keep it short and effective.
- Each and every component in the document should be written by keeping only the Business Requirements in mind.
- Reliable and adaptable – It should be updatable as required in the future releases.
- Consistent – It should not have more changes in the future.
- Follow the template provided by the Organization or Customer.
Acceptance Test Plan Template
Here we will take a look at a common template for Acceptance Test Plan which can be further tweaked as per the project requirements.
<Mention Project Name with its release version if required. Say Project_MainRelease1.0>
<Mention the objective of the project. Explain the purpose of the project and also mention the category of the end-users to whom the project is intended to. For Example: If the project is to serve service for Bank employees/Sales Person/Educational Institute etc., then it has to be clearly mentioned. Also, it should mention the reason for which that particular requirement has been chosen.>
Revision History/Change Log
<This should be in a tabular form with the below information:
- Date – The date on which the document was modified.
- Modified By – Who has changed the contents of the document.
- Purpose – Why was the document modified.
- Version – Current Version of the document after modifications (goes as 1.0, 1.1, 1.2, 1.3,… for a particular release. Next release will start from 2, 2.1, 2.2, 2.3,…, The list goes on).
- Approved By – Who has approved the changes made (implicitly means that document has been reviewed and approved).
The very first row in this table should be the document created details. Then follows the details of the changes made.>
Table of Contents
<Mention each and every component along with its page number.>
<Provide references to relevant documents for testing: Requirement Specification Document, Use Cases, etc.>
<Mention the scope of testing (Example: All the business scenarios).>
<Clearly explain how the acceptance testing phase has reached in the project starting from the development. Provide details about the actions performed in this phase and how it was done. Also, the different types of testing performed in this phase can be detailed like Positive testing, Negative testing (not always), etc.>
<Mention all the test items for this phase: Features/modules/sub-modules etc. (Example: Client Application, Server Application, etc.)>
Features to be Tested
<Mention all the features that need to be tested: Basic functionalities (Example: Search by different keywords, navigations between modules and sub-modules, etc.), Security features (Example: Registration to website, login to website, transactions, etc.)>
Features not to be Tested
<Explicitly mention all the features that are not to be tested like Resilience to server, database crashes, network delays, etc. >
<Mention the approach to test in this phase. Provide details on how the testing is performed (Example: Once all the acceptance tests are executed, exploratory testing will be performed for a very short duration to assess the ease and use of the product. Each and every bug encountered will be logged in the bug management tool and will be discussed in bug triage meetings for RCA's…)>
Test Environment Details
<Mention testing environment details: Staging/UAT/Pre-Prod/ etc., Also mention Test Bed set up information (Refer Acceptance Testing – Part 1).>
<Mention all the entry criteria for Acceptance Testing to begin. Refer Acceptance Testing – Part1 for more details.>
Tests – If there are no separate Acceptance tests written
<List the series of tests to be conducted with each step described in detail.
Each test must include:
- Test #.
- A description of what is being tested (Example: Verify whether a user is able to create an account successfully).
- Business requirement to which this test maps (Traceability Matrix) – Very important.
- State of the product before starting the testing (User should be registered successfully but not activated the account, User should have accessed the product at least 30 days ago, etc.)
- Any server conditions – Should the server be down for some time.
- Test Steps: Detailed numbered flow (Example: see below
- Open the application.
- Attempt to login with valid credentials with Remember Me checkbox selected).
- Expected Result: What is the expected behavior of the step>
Acceptance Tests – If there are separate Acceptance tests written
<Provide the link to Acceptance Tests document.>
<Mention all the exit criteria for Acceptance Testing to end. Refer Acceptance Testing – Part1 for more details.>
<Mention the names of all the members who will be a part of the Acceptance Testing phase in a proper order. Also, mention their role in the phase.>
Roles and Responsibilities
<Mention the responsibilities that each role has to perform.>
<Mention what tools have to be used, maybe for bug logging, test management to update results, etc., Provide proper references to each of the tools which will guide the resources on how to use them.>
Business Decision Factors
<Mention all the factors which lead to business decision either to Go or No-Go.>
<Clearly describe how the Acceptance testing Sign-off takes place and who is responsible for signing off.>
Point of Contact
<Mention the point of contact for the Accepting Testing Phase with the Name, Corporate Email Address, Role and Contact Number.>
Acceptance test plan is considered as the Master Test Plan for the Phase.
Reviewing Acceptance Test Plan
Once the plan is ready, it has to be reviewed for completeness, non-ambiguity, clearness, quality, etc. No doubt that, the entire content in the Acceptance test plan has to be reviewed thoroughly for proper information, but, it has to be reviewed against few other points as well let’s say checklist points.
Here, let’s categorize the contents and see the checklist points against them.
|Title||Is the title matching the project title as referred everywhere|
Is the title following Project’s naming conventions
|Revision History, Table of Contents||Is every version modifications tracked properly for the plan|
Has every version change undergone proper review and is mentioned
Is the versioning convention correct
Does the table of contents match the actual contents in the plan
Is the page number for each content correct
Is the page number updated if the modifications made in the plan changed page number of the contents
|References||Are the references existing and valid|
Do they match with scope
Are they complete and considered for tests identification
|Test Items, Features to be tested, Features not to be tested||Are they numbered|
Does each feature / module / sub-module comes under scope
Can the schedule planned cover all the identified test items within
|Entry Criteria, Exit Criteria||Are they numbered|
Is each and every criteria mentioned in detail
|Test Environment Details||Is it having all the required configurations mentioned|
Is the version for each configuration specific or latest to be considered
Do the VMs, environment exists (if not, mention possible date for its availability)
Is the credentials sharing method for particular environment access mentioned
|Acceptance Tests||Are the tests numbered|
Are the Preconditions numbered
Are the test steps clear to understand
Are the test steps complete
Is the expected result complete
Is there any open question in the tests (if any, follow-up and complete it)
Is the reference to Acceptance Tests (if written separately) valid and existing
Is the traceability correct
Is there any business requirement missed out to cover for test
|Resources, Roles, and Responsibilities||Are the responsibilities for each role numbered|
Can the responsibilities be achieved
Is the identified resource capable of handling mentioned responsibilities
|Tools||Are all the tools mentioned|
Are all the tools numbered
Are all the tools versioned
Does any of the tool need license or the existing license valid during the phase
Is the guidance to the tool usage correct and sufficient
|Business Decision Factors||Has all the factors mentioned|
Are all the factors numbered
|Sign-Off Procedure||Is the procedure valid|
Is the procedure acceptable
Is the procedure clear to understand
|Point of Contact||Is the resource identified as point of contact available in the organization during the phase|
Is the resource identified capable of handling the phase
Any test plan satisfying the above checklist document will serve as a strong document for internal audits as well.
Acceptance tests were earlier known as Functional tests. In order to make the name more suitable for Acceptance testing phase and to serve the purpose, it was renamed as Acceptance Tests. Sometimes it is also termed as Customer Tests.
Acceptance tests are always derived from user stories, acceptance criteria, and use cases. These are black-box system tests and represent only those business tests that have to be verified. These should be intended mainly for product behavior, usage, and flows.
The designed acceptance tests can also be taken into consideration for system testing phase in the regression cycles to gain confidence on the product before handing it over to acceptance testing phase.
Key points to be remembered before writing acceptance tests:
- Keep all the reference documents in place: Software Requirement Specification, Business Requirements Document, Use Cases, User Stories, Data matrix (in case of logic involved) etc.
- Focus only on business requirements (testable business requirements).
- Clear all the doubts, queries on the business requirements at the earliest.
- Make sure there are no changes on the requirements for the current release at least.
General and Simple Template to write Acceptance Tests:
This template can again be tweaked as per the project needs and with more information to include.
Now, let’s take some common scenarios and see how Acceptance test scenarios can be written on them.
Case 1: User Account handling
This is the scenario where the users are allowed to Create, View, Update, and De-activate their account. In general, it’s a CRUD operation (Create, Read, Update, and Delete). So directly we will get 4 major scenarios to test.
Along with this, in real-time user account handling, we have many areas when it comes to viewing and updating.
Proceeding with writing acceptance tests:
Test 1: Registration/Sign Up/Create Account, verify if a User is able to:
- Create the Account.
- Activate the account.
- Activate the account only once (Here, activation link has to be tested for 2nd Even though this is negative testing, it is one of the major verification points to be considered).
Test 2: To Access and View Account information, verify if a User is able to:
- Log in to the account.
- View different sections in the Profile (If Profile section is categorized, then each and every category should be viewable).
- Verify that the data displayed in Profile is correct as per the User’s input.
Test 3: To Update Account Information, verify if a User is able to:
- Update Account Information (Profile):
- Update each and every category of the Profile.
- Verify that the update information is reflected correctly in the Profile.
- Verify if the User is not able to update information in the Profile (In some applications, First name, Last name, Username, etc. will not be allowed to update. Even though this is negative testing, it is one of the major verification points to be considered).
- Cancel the Update flow (Even though this is negative testing, it is also one of the major verification points to be considered).
Test 4: If De-activation of account is allowed, then, verify if a User is able to:
- De-activate the account.
- Cancel de-activation flow (Even though this is negative testing, it is one of the major verification points to be considered).
- Access account after canceling de-activation.
Test 5: If verifications are required for an Email address or Phone numbers, then, verify if a User is able to:
- Update Email address to the other valid one.
- Verify” updated Email address.
- Verify if updated and “verified” Email address is considered further – Send some emails from the application and check for its arrival to the updated Email address. The old one should not receive Emails.
- Add the new Phone number.
- Verify added phone number through Call.
- Verify added phone number through SMS.
- Verify that the added and “verified” phone number is reflecting in the account.
- Update the Phone number.
- Verify updated phone number through Call.
- Verify ” updated phone number through SMS.
- Verify if updated and “verified” phone number is reflecting in the account.
Case 2: Purchasing Product
Purchasing of the product usually has the general flow.
Some general scenarios what end-users look at are listed here:
Pre-condition: User should be logged in to the application.
Test 1: Product Details, verify if a User is able to:
- View the Product details page.
- View all the sub-sections in the Product details page (Description, Feature, Brand information, etc.).
- Select the Quantity of the product, Color, Size, etc. as available in the Product details page.
- Navigate to the category, sub-category pages from the Product Details page (if available in Product details page).
- Navigate to the other Product’s details page (if provided relevant products section).
- View comments and ratings on the product.
- Sort Comments of the Product based on ratings.
- View overall rating of the Product.
- Add Comment on the Product.
- Update his/her comment on the product.
- Delete his/her comment on the product (if provided).
Test 2: Add to Cart, verify if a User is:
- Able to add the product to Cart:
- Through Product details page.
- Through Product list page.
- Able to add required quantity to the cart ( 1 to max limit set).
- Not able to add the product to the Cart if Out-of-Stock.
Test 3: In the Cart Page, verify if a User is able to:
- View the Product in the Cart with Price details for added quantity.
- Update quantity (1 to max limit set).
- Remove the Product from Cart.
- Navigate back to shopping.
- Continue to Checkout.
- View Empty Cart when no product is added,
Test 4: In the Account Details Page, verify if a User is able to:
- Continue with the existing Shipping details.
- Update Shipping address.
- Add new shipping address.
- Continue with the existing Phone number.
- Update Phone number for the order.
- Add new Phone number for the order.
- Navigate back to the Cart page.
- Navigate to the Payment page.
Test 5: On the Payments Page, verify if a User is able to:
- Verify the correctness of the amount to be billed.
- Process the order with all the available options (One option for each separate order).
- Process transaction Successfully. Go to the Order Confirmation page.
- Transaction Failure (Even though this is negative testing, it should be considered as a major scenario).
- Apply coupons:
- Valid Coupons – Success. Here verify the change in the amount to be billed.
- Invalid coupons – Failure
- Expired coupons – Failure.
- Navigate back to the Account details page.
Reviewing Acceptance Tests
Reviewing Acceptance tests is an important task as it needs to be correct and to-the-point with respect to the business requirements. As these may be conducted by Customers themselves and/or end-users, it is very much necessary to be complete, non-ambiguous, correct, and detailed enough for anyone to understand and execute.
Reviewing Acceptance tests has to be done by Business Analysts, Customers and any review comments should be incorporated into high priority.
At the individual test level, the review should be done against the below:
- Whether test covers the business requirement or not.
- Are the pre-conditions clear?
- Are the test steps easy to understand and detailed?
- Is the expected result correct and clear?
- Is it mapped to the Business requirements for traceability?
- Is the test complete enough to cover the particular flow or usage?
- Is the particular test required as part of acceptance testing.
- Is there some verification point that is not needed for acceptance testing.
- Is it purely functional or any GUI is covered within (It should be only Functional).
- Is the special input data necessary? If yes, is it provided for particulars?
On the whole, entire Acceptance tests suite review should cover:
- Bi-directional traceability: Business requirements to Tests AND Tests to Business requirements.
- Is each and every business requirement covered?
- Is every business requirement covered by one or more tests?
- Are the business rules covered?
- Is the special data case handled?
- How many tests are written to cover each requirement or rule?
- Can the tests be grouped together and classified for flows.
- Are the tests sequenced properly so that the execution is efficient?
In a nutshell, as mentioned earlier, documents play a very drastic role in Acceptance testing.
Hence, any acceptance test that is written should be well-structured and in-flow with its usage, so that it keeps the acceptance testers to be interested in what they are testing and how they are doing it. This, in turn, would automatically bring in success.
Stay tuned and watch out the upcoming Acceptance Testing tutorial to know more about Acceptance Testing Reports along with some generic templates. Also, let us know if you have any queries.