Introduction to Acceptance Testing (Part-I):
In this tutorial series, you will learn:
- What is Acceptance Testing
- Acceptance Tests and Test Plan
- Acceptance Tests Status and Summary Reports
- What is User Acceptance Testing (UAT)
Are you done with System Testing? Are most of your bugs fixed? Are the bugs verified and closed? So, what's next?
Next in the list comes Acceptance Testing, which is the last phase of the Software Testing Process. This is the phase where the customer decides GO/No-GO for the product and has to be compulsorily followed before releasing Product to the market. Joint efforts of the development and the testing team will be awarded by the customer by either accepting or rejecting the Product developed.
This unique tutorial on Acceptance Testing will give you a complete overview of the meaning, types, uses and various other factors involved in Acceptance Test in a simple and easy manner for your better understanding.
What You Will Learn:
- What is Acceptance Testing?
- Why Acceptance Tests?
- Who does Acceptance Testing?
- Qualities of Acceptance Testers
- Differences between System Testing, Acceptance Testing, and User Acceptance Testing
- Acceptance Tests
- Acceptance Test Bed
- Entry and Exit Criteria for AT
- Acceptance Testing Process
- Success factors for This Testing
- Recommended Reading
What is Acceptance Testing?
Once the System Testing process is completed by the testing team and is signed-off, the entire Product/application is handed over to the customer/few users of customers/both, to test for its acceptability i.e., Product/application should be flawless in meeting both the critical and major Business requirements. Also, end-to-end business flows are verified similar as in real-time scenario.
The production-like environment will be the testing environment for Accepting Testing (Usually termed as Staging, Pre-Prod, Fail-Over, UAT environment).
This is a black-box testing technique where only the functionality is verified to ensure that the product meets the specified acceptance criteria (no need for design/implementation knowledge).
Why Acceptance Tests?
Though System testing has been completed successfully, the Acceptance test is demanded by the customer. Tests conducted here are repetitive, as they would have been covered in System testing.
Then, why is this testing is conducted by customers?
This is because:
- To gain confidence in the product that is getting released to the market.
- To ensure that the product is working in the way it has to.
- To ensure that the product matches current market standards and is competitive enough with the other similar products in the market.
There are several types of this testing.
Few of them are listed below:
#1) User Acceptance Testing (UAT)
UAT is to assess whether the Product is working for the user, correctly for the usage. Specific requirements which are quite often used by the end-users are primarily picked for the testing purpose. This is also termed as End-User Testing.
The term “User” here signifies the end-users to whom the Product/application is intended and hence, testing is performed from the end-users perspective and from their point of view.
=> Also Read: What is User Acceptance Testing (UAT)?
#2) Business Acceptance Testing (BAT)
This is to assess whether the Product meets the business goals and purposes or not.
BAT mainly focuses on business benefits (finances) which are quite challenging due to the changing market conditions/advancing technologies so that the current implementation may have to undergo changes which result in extra budgets.
Even the Product passing the technical requirements may fail BAT due to these reasons.
#3) Contract Acceptance Testing (CAT)
This is a contract which specifies that once the Product goes live, within a predetermined period, the acceptance test must be performed and it should pass all the acceptance use cases.
Contract signed here is termed as Service Level Agreement (SLA), which includes the terms where the payment will be made only if the Product services are in-line with all the requirements, which means the contract is fulfilled.
Sometimes, this contract may happen before the Product goes live. Either the ways, a contract should be well defined in terms of the period of testing, areas of testing, conditions on issues encountered at later stages, payments, etc.
#4) Regulations/Compliance Acceptance Testing (RAT)
This is to assess whether the Product violates the rules and regulations that are defined by the government of the country where it is being released. This may be unintentional but will impact negatively on the business.
Usually, the developed Product/application that is intended to be released all over the world, has to undergo RAT, as different countries/regions have different rules and regulations defined by its governing bodies.
If any of the rules and regulations are violated for any country, then that country or the specific region in that country will not be allowed to use the Product and is considered as a Failure. Vendors of the Product will be directly responsible if the Product is released even though there is a violation.
#5) Operational Acceptance Testing (OAT)
This is to assess the operational readiness of the Product and is a non-functional testing. It mainly includes testing of recovery, compatibility, maintainability, technical support availability, reliability, fail-over, localization etc.
OAT mainly assures the stability of the Product before releasing it to the production.
#6) Alpha Testing
This is to assess the Product in the development/testing environment by a specialized testers team usually called alpha testers. Here, the testers feedback, suggestions help to improve the Product usage and also to fix certain bugs.
Here, testing happens in a controlled manner.
=> Also Read: What is Alpha Testing?
#7) Beta Testing/Field Testing
This is to assess the Product by exposing it to the real end-users, usually called beta testers/beta users, in their environment. Continuous feedback from the users is collected and the issues are fixed. Also, this helps in enhancing/improving the Product to give a rich user experience.
Testing happens in an uncontrolled manner, which means a user has no restrictions on the way in which the Product is being used.
=> Also Read: What is Beta Testing?
All these types have a common goal:
- Ensure to gain/enrich Confidence in the Product.
- Ensure that the Product is ready to be used by the real users.
Who does Acceptance Testing?
For Alpha type, only the members of the organization (who developed the Product) perform the testing. These members are not directly a part of the project (Project managers/leads, developers, testers). Management, Sales, Support teams usually perform the testing and provide feedback accordingly.
Apart from the Alpha type, all other acceptance types are generally performed by different stakeholders. Like customers, customer’s customers, specialized testers from the organization (not always).
It is also good to involve Business Analysts and Subject Matter Expertise while performing this testing based on its type.
Qualities of Acceptance Testers
Testers with the below qualities are qualified as Acceptance testers:
- Ability to think logically and analytically.
- Good domain knowledge.
- Able to study the competitive products in the market and analyze the same in the developed product.
- Having end-user perception while testing.
- Understand business need for each requirement and test accordingly.
Impact of Issues found during this testing
Any issues encountered in Acceptance test phase should be considered as a high priority one and fixed immediately. This also requires Root Cause Analysis to be performed on each and every issue that is found.
The testing team plays a major role in providing RCA's for Acceptance issues. These also help in determining how efficiently testing is performed.
Also, valid issues in acceptance test will hit both the testing and the development team efforts in terms of impression, ratings, customer surveys, etc. Sometimes, if any ignorance from the testing team on validations is found, it leads to escalations as well.
This testing is useful from several aspects.
Few of which include:
- To figure out the issues missed during the functional testing phase.
- How well the Product is developed.
- A product is what actually the customers need.
- Feedbacks/surveys conducted help in improving the Product performance and user experience.
- Improve the process followed by having RCAs as input.
- Minimize or eliminate the issues arising from the Production Product.
Differences between System Testing, Acceptance Testing, and User Acceptance Testing
Given below are the prime differences between these 3 types of Acceptance tests.
|Acceptance Testing||User Acceptance Testing|
|End-to-end testing is performed to verify whether Product meets all the specified requirements||Testing is performed to verify whether Product meets customer requirements for acceptability||Testing is performed to verify whether end-users requirements are fulfilled for acceptability|
|A product is tested as the whole focusing only on functional and non-functional needs||Product is tested for business needs – user acceptability, business goals, rules and regulations, operations, etc.||Product is tested only for user acceptability|
|Testing team performs System Testing||Customer, Customers’ customers, tester (rarely), management, Sales, Support teams performs acceptance testing depending on the type of test carried out||Customer, Customers’ customer, testers (rarely) performs user acceptance testing|
|Test cases are written and executed||Acceptance tests are written and executed||User Acceptance tests are written and executed|
|Can be functional and non-functional||Usually Functional, but non-functional in case of RAT, OAT, etc||Only Functional|
|Only test data is used for testing||Real-time data/production data is used for testing||Real-time data / Production data is used for testing|
|Positive and negative tests are performed||Usually Positive tests are performed||Only Positive tests are performed|
|Issues found are considered as bugs and fixed based on severity and priority||Issues found marks Product as Failure, and considered to be fixed immediately||Issues found marks Product as Failure and considered to be fixed immediately|
|Controlled manner of testing||Can be controlled or uncontrolled based on type of testing||Uncontrolled manner of testing|
|Testing on Development environment||Testing on Development environment or pre-production environment or production environment, based on type||Testing is always on Pre-Production environment|
|No assumptions, but if any can be communicated||No assumptions||No assumptions|
Similar to Product test cases, we do have acceptance tests. Acceptance tests are derived from User stories’ acceptance criteria. These are usually the scenarios that are written at the high-level detailing on what the Product has to do under different conditions.
It does not give a clear picture on how to perform tests, as in test cases. Acceptance tests are written by Testers who have a complete grip on the Product, usually Subject Matter Expertise. All the tests written are reviewed by a customer and/or business analysts.
These tests executed during acceptance test. Along with acceptance tests, a detailed document on any set-ups to be done has to be prepared. It should include every minute details with proper screenshots, set-up values, conditions, etc.
Acceptance Test Bed
Test Bed for this testing is similar to a regular test bed but is a separate one. Platform with all the required hardware, software, operating products, network set-up & configurations, server set-up & configurations, database set-up & configurations, licenses, plug-ins, etc., have to be set up very much alike the Production environment.
Acceptance test bed is a platform/environment where the designed acceptance tests will be executed. Before handing over the Acceptance test environment to the customer, it is a good practice to check for any environmental issues and stability of the Product.
If there is no separate environment set up for acceptance testing, a regular testing environment can be used for that purpose. But here, it will be messy as the test data from regular System Testing, and the real-time data from acceptance testing are maintained in a single environment.
Acceptance test bed is usually set up on the customer-side (i.e., in the laboratory) and will have restricted access to the development and testing teams.
Teams will be required to access this environment through VMs/or specifically designed URLs using special access credentials, and all the access to this will be tracked. Nothing on this environment has to be added/modified/deleted without the customer’s permission, and they should be notified of the changes that are made.
Entry and Exit Criteria for AT
Just as any other phase in the STLC, Acceptance testing does have a set of entry and exit criteria which are to be well-defined in Acceptance Test Plan (which is covered in the later part of this tutorial).
This is the phase which starts right after System testing and ends before the Production launch. So, the Exit criteria of System testing becomes a part of the Entry criteria for AT. Similarly, the Exit criteria of AT become a part of Entry criteria for the Production Launch.
Given below are the conditions to be fulfilled before starting:
- Business requirements should be clear and available.
- System and Regression testing phase should be completed.
- All the Critical, Major & Normal bugs should be fixed and closed (Minor bugs accepted mainly are cosmetic bugs that do not disturb the usage of the product).
- Known issues list should be prepared and shared with the stakeholders.
- Acceptance Test Bed should be set up and high-level check should be performed for no environmental issues.
- System Testing phase should be signed-off letting the product to move to the AT phase (Usually done through Email communication).
There are certain conditions to be fulfilled by AT to let the product go for a Production Launch.
They are as follows:
- Acceptance tests should be executed and all the tests should Pass.
- No Critical/Major defects left Open. All the defects should be fixed and verified immediately.
- AT should be Signed-off-by all the included stakeholders with Go/No-Go Decision on the product.
Acceptance Testing Process
In V-Model, AT phase is in parallel to the Requirements phase.
Actual AT process goes as shown below:
Business Requirements Analysis
Business requirements are analyzed by referring all the available documents within the project.
Some of which are:
- System Requirement Specifications
- Business Requirements Document
- Use Cases
- Workflow diagrams
- Designed data matrix
Design Acceptance Test Plan
There are certain items to be documented in the Acceptance Test Plan.
Let's take a look at some of them:
- Acceptance Testing strategy and approach.
- Entry and exit criteria should be well-defined.
- The scope of AT should be well-mentioned and it has to cover only the business requirements.
- Acceptance test design approach should be detailed so as anyone writing tests can easily understand the way in which it has to be written.
- Test Bed set up, actual testing schedule/timelines should be mentioned.
- As testing is conducted by different stakeholders, details on logging bug should be mentioned as the stakeholders may not be aware of the procedure followed.
Design and Review Acceptance Tests
Acceptance tests should be written at a scenario level mentioning what has to be done (not in-detail to include how to do). These should be written only for the identified areas of scope for business requirements, and each and every test has to be mapped to its referencing requirement.
All the written acceptance tests have to be reviewed to achieve high coverage on business requirements.
This is to make sure that any other tests apart from scope mentioned are not involved so that testing lies within the scheduled timelines.
Acceptance Test Bed Set up
Test Bed should be set up similar to a Production environment. Very high-level checks are required to confirm on environment stability and usage. Share the credentials to use the environment only with a stakeholder who is performing this testing.
Acceptance Test Data Set Up
Production data has to be prepared/populated as test data in the systems. Also, there should be a detailed document in such a way that the data has to be used for testing.
Do not have the test data like TestName1, TestCity1, etc., Instead have Albert, Mexico, etc. This gives a rich experience of real-time data and testing will be up-to-the-point.
Acceptance Test Execution
Designed Acceptance tests have to be executed on the environment at this step. Ideally, all the tests should pass at the first attempt itself. There should be no functional bugs arising out of Acceptance testing, if any, then they should be reported at a high priority to be fixed.
Again, bugs fixed have to be verified and closed as a high priority task. Test execution report has to be shared on a daily basis.
Bugs logged in this phase should be discussed in a bug-triage meeting and has to undergo Root Cause Analysis procedure. This is the only point where acceptance testing assess whether all the business requirements are actually met by the product or not.
There comes out a Go/No-Go decision for the product to be launched in Production. Go decision will take the product ahead to be released to the market. No-Go decision marks the product as Failure.
Few factors of No-Go Decision:
- Poor Quality of the product.
- Too Many open Functional Bugs.
- Deviation from business requirements.
- Not up to the market standards and needs enhancements to match the current market standards.
Success factors for This Testing
Once this test is planned, prepare a checklist which increases the success rate of it. There are some action items that are to be followed before Acceptance test starts.
- Have a well-defined scope and make sure there is a business need for the scope identified for this testing.
- Execute Acceptance tests in System testing phase itself at least once.
- Perform extensive ad-hoc testing for each of the acceptance test scenarios.
In a nutshell, Acceptance testing helps in figuring out the efficiency of development and testing teams.
There are several tools to conduct this activity, but usually, it is preferred to be done manually as there is an involvement of the real-users and different stakeholders who are not from a technical background, and it may not be feasible for them.
In our next tutorial, we will hover on the below topics:
- Acceptance test criteria examples.
- How to write an Acceptance Test Plan.
- A suitable template for Acceptance Tests writing.
- How to write Acceptance tests with examples.
- Identifying Acceptance Test scenarios.
- Acceptance test reports.
- Acceptance testing in Agile and test-driven development.
Have you performed Acceptance Testing? We would be glad to hear your experiences!!