What is Acceptance Testing (A Complete Guide)

Introduction to Acceptance Testing (Part-I):

In this tutorial series, you will learn:

1. What is Acceptance Testing
2. Acceptance Tests and Test Plans
3. Acceptance Test Status and Summary Reports
4. What is User Acceptance Testing (UAT)

Are you done with System Testing? Have most of your bugs been fixed? Are the bugs verified and closed? So, what’s next?

Next on the list comes Acceptance Testing, which is the last phase of the Software Testing Process. This is the phase where the customer decides GO/No-GO for the product and has to be compulsorily followed before releasing the Product to the market. Joint efforts of the development and testing team will be awarded by the customer by either accepting or rejecting the Product developed.

Acceptance Testing

This unique tutorial on Acceptance Testing will give you a complete overview of the meaning, types, uses, and various other factors involved in Acceptance Tests in a simple and easy manner for your better understanding.

What is Acceptance Testing?

Once the System Testing process is completed by the testing team and is signed-off, the entire Product/application is handed over to the customer/few users of customers/both, to test for its acceptability i.e., Product/application should be flawless in meeting both the critical and major Business requirements. Also, end-to-end business flows are verified similar to in real-time scenarios.

The production-like environment will be the testing environment for Acceptance Testing (Usually termed as Staging, Pre-Prod, Fail-Over, UAT environment).

This is a black-box testing technique where only the functionality is verified to ensure that the product meets the specified acceptance criteria (no need for design/implementation knowledge).

Why Acceptance Tests?

Though System testing has been completed successfully, the Acceptance test is demanded by the customer. Tests conducted here are repetitive, as they would have been covered in System testing.

So why is this testing being conducted by customers?

This is because:

• To gain confidence in the product that is getting released to the market.
• This is to ensure that the product is working the way it has to.
• This is to ensure that the product matches current market standards and is competitive enough with other similar products in the market.

Types

There are several types of this testing.

A few of them are listed below:

#1) User Acceptance Testing (UAT)

UAT is to assess whether the Product is working for the user, correctly for the usage. Specific requirements which are quite often used by the end-users are primarily picked for testing purposes. This is also termed End-User Testing.

The term “User” here signifies the end-users to whom the Product/application is intended and hence, testing is performed from the end-users perspective and from their point of view.

=> Also Read: What is User Acceptance Testing (UAT)?

#2) Business Acceptance Testing (BAT)

This is to assess whether the Product meets the business goals and purposes or not.

BAT mainly focuses on business benefits (finances) which are quite challenging due to the changing market conditions/advancing technologies so the current implementation may have to undergo changes that result in extra budgets.

Even the Product passing the technical requirements may fail BAT due to these reasons.

#3) Contract Acceptance Testing (CAT)

This is a contract that specifies that once the Product goes live, within a predetermined period, the acceptance test must be performed and it should pass all the acceptance use cases.

Contract signed here is termed a Service Level Agreement (SLA), which includes the terms where the payment will be made only if the Product services are in-line with all the requirements, which means the contract is fulfilled.

Sometimes, this contract may happen before the Product goes live. Either way, a contract should be well defined in terms of the period of testing, areas of testing, conditions on issues encountered at later stages, payments, etc.

#4) Regulations/Compliance Acceptance Testing (RAT)

This is to assess whether the Product violates the rules and regulations that are defined by the government of the country where it is being released. This may be unintentional but will negatively impact the business.

Usually, the developed Product/application that is intended to be released all over the world, has to undergo RAT, as different countries/regions have different rules and regulations defined by their governing bodies.

If any of the rules and regulations are violated by any country, then that country or a specific region in that country will not be allowed to use the Product and is considered a Failure. Vendors of the Product will be directly responsible if the Product is released even if there is a violation.

#5) Operational Acceptance Testing (OAT)

This is to assess the operational readiness of the Product and is non-functional testing. It mainly includes testing of recovery, compatibility, maintainability, technical support availability, reliability, fail-over, localization, etc.

OAT mainly assures the stability of the product before releasing it to production.

#6) Alpha Testing

This is to assess the Product in the development/testing environment by a specialized tester team usually called alpha testers. Here’s the tester’s feedback and suggestions to help improve Product usage and also to fix certain bugs.

Here, testing happens in a controlled manner.

=> Also Read: What is Alpha Testing?

#7) Beta Testing/Field Testing

This is to assess the Product by exposing it to real end-users, usually called beta testers/beta users, in their environment. Continuous feedback from the users is collected and the issues are fixed. Also, this helps in enhancing/improving the Product to give a rich user experience.

Testing happens in an uncontrolled manner, which means a user has no restrictions on the way in which the Product is being used.

=> Also Read: What is Beta Testing?

All these types have a common goal:

• Ensure to gain/enrich Confidence in the Product.
• Ensure that the Product is ready to be used by real users.

Who does Acceptance Testing?

For the Alpha type, only the members of the organization (who developed the Product) perform the testing. These members are not directly a part of the project (Project managers/leads, developers, testers). Management, Sales, and Support teams usually perform the testing and provide feedback accordingly.

Apart from the Alpha type, all other acceptance types are generally performed by different stakeholders. Like customers, customer’s customers, specialized testers from the organization (not always).

It is also good to involve Business Analysts and Subject Matter Expertise while performing this testing based on its type.

Qualities of Acceptance Testers

Testers with the below qualities are qualified as Acceptance testers:

• Ability to think logically and analytically.
• Good domain knowledge.
• Able to study the competitive products in the market and analyze the same in the developed product.
• Having an end-user perception while testing.
• Understand the business needs for each requirement and test accordingly.

Impact of Issues found during this testing

Any issues encountered in the Acceptance test phase should be considered a high priority and fixed immediately. This also requires Root Cause Analysis to be performed on each and every issue that is found.

The testing team plays a major role in providing RCA’s for Acceptance issues. This also helps in determining how efficient testing is performed.

Also, valid issues in the acceptance test will hit both the testing and development team efforts in terms of impressions, ratings, customer surveys, etc. Sometimes, if any ignorance from the testing team on validations is found, it leads to escalations as well.

Use

This testing is useful in several aspects.

A few of these include:

• Figure out the issues missed during the functional testing phase.
• How well the product is developed.
• A product is what the customers need.
• Feedback/surveys conducted help in improving product performance and user experience.
• Improve the process followed by having RCAs as input.
• Minimise or eliminate issues arising from the Production Product.

Differences between System Testing, Acceptance Testing, and User Acceptance Testing

Given below are the prime differences between these 3 types of Acceptance tests.

Acceptance Tests

Similar to Product test cases, we do have acceptance tests. Acceptance tests are derived from User Stories’ acceptance criteria. These are usually the scenarios that are written at a high-level detailing what the Product has to do under different conditions.

It does not give a clear picture of how to perform tests, as in test cases. Acceptance tests are written by Testers who have a complete grip on the Product, usually Subject Matter Expertise. All the tests written are reviewed by the customer and/or business analysts.

These tests are executed during the acceptance test. Along with acceptance tests, a detailed document on any set-ups to be done has to be prepared. It should include every minute details with proper screenshots, set-up values, conditions, etc.

Acceptance Test Bed

The test bed for this testing is similar to a regular testbed but is a separate one. Platforms with all the required hardware, software, operating products, network set-up & configurations, server set-up & configurations, database set-up & configurations, licenses, plug-ins, etc., have to be set up very much like the Production environment.

The acceptance testbed is a platform/environment where the designed acceptance tests will be executed. Before handing over the Acceptance test environment to the customer, it is good practice to check for any environmental issues and stability of the Product.

If there is no separate environment set up for acceptance testing, a regular testing environment can be used for that purpose. But here, it will be messy as the test data from regular System Testing and the real-time data from acceptance testing are maintained in a single environment.

The acceptance testbed is usually set up on the customer-side (i.e., in the laboratory) and will have restricted access to the development and testing teams.

Teams will be required to access this environment through VMs/or specifically designed URLs using special access credentials, and all access to this will be tracked. Nothing in this environment has to be added/modified/deleted without the customer’s permission, and they should be notified of the changes that are made.

Entry and Exit Criteria for AT

Just like any other phase in the STLC, Acceptance testing does have a set of entry and exit criteria which are to be well-defined in the Acceptance Test Plan (which is covered in the latter part of this tutorial).

This is the phase that starts right after System testing and ends before the Production launch. As a result, the Exit criteria for System testing has become a part of the Entry criteria for AT. Similarly, the Exit criteria for AT has become part of the Entry criteria for the Production Launch.

Entry Criteria

Given below are the conditions to be fulfilled before starting:

• Business requirements should be clear and available.
• The system and Regression testing phase should be completed.
• All the Critical, Major & Normal bugs should be fixed and closed (Minor bugs accepted mainly are cosmetic bugs that do not disturb the usage of the product).
• Known issues list should be prepared and shared with the stakeholders.
• Acceptance Test Beds should be set up and a high-level check should be performed for no environmental issues.
• The System Testing phase should be signed-off letting the product move to the AT phase (Usually done through email communication).

Exit Criteria

There are certain conditions that need to be fulfilled by AT to allow the product to go for a Production Launch.

They are as follows:

• Acceptance tests should be executed and all the tests should pass.
• No Critical/Major defects left Open. All defects should be fixed and verified immediately.
• AT should be Signed-off-by all the included stakeholders with a Go/No-Go Decision on the product.

Acceptance Testing Process

In V-Model, the AT phase is in parallel to the Requirements phase.

The actual AT process goes as shown below:

Business Requirements Analysis

Business requirements are analyzed by referring to all the available documents within the project.

Some of which are:

• System Requirement Specifications
• Business Requirements Document
• Use Cases
• Workflow diagrams
• Designed data matrix

Design Acceptance Test Plan

There are certain items that need to be documented in the Acceptance Test Plan.

Let’s take a look at some of them:

• Acceptance Testing strategy and approach.
• Entry and exit criteria should be well-defined.
• The scope of AT should be well-mentioned and it has to cover only the business requirements.
• The acceptance test design approach should be detailed so that anyone writing tests can easily understand the way in which it has to be written.
• Test Bed set up and actual testing schedule/timelines should be mentioned.
• As testing is conducted by different stakeholders, details on logging bugs should be mentioned as the stakeholders may not be aware of the procedure followed.

Design and Review Acceptance Tests

Acceptance tests should be written at a scenario level mentioning what has to be done (not in detail to include how to do it). These should be written only for the identified areas of scope for business requirements, and each and every test has to be mapped to its referencing requirement.

All written acceptance tests have to be reviewed to achieve high coverage of business requirements.

This is to ensure that all other tests apart from the scope mentioned are not involved so that testing lies within the scheduled timelines.

Acceptance Test Bed Set up

Test Beds should be set up similar to a Production environment. Very high-level checks are required to confirm environment stability and usage. Share your credentials to use the environment only with the stakeholder who is performing this testing.

Acceptance Test Data Set-Up

Production data has to be prepared/populated as test data in the systems. Also, there should be a detailed document in such a way that the data has to be used for testing.

Do not have the test data like TestName1, TestCity1, etc., instead I have Albert, Mexico, etc. This gives a rich real-time data experience and testing will be up-to-the-point.

Acceptance Test Execution

Designed Acceptance tests have to be executed in the environment at this step. Ideally, all the tests should pass at the first attempt itself. There should be no functional bugs arising out of Acceptance testing, if any, then they should be reported as a high priority to be fixed.

Again, bug fixes have to be verified and closed as a high-priority task. Test execution reports have to be shared on a daily basis.

Bugs logged in this phase should be discussed in a bug-triage meeting and have to undergo the Root Cause Analysis procedure. This is the only point where acceptance testing assesses whether all the business requirements are met by the product or not.

Business Decision

A go/no-go decision has come out for the product to be launched in Production. Go decision will take the product ahead to be released to the market. A no-go decision marks the product as a failure.

A few factors related to the No-Go Decision:

• Poor quality of the product.
• Too Many open Functional Bugs.
• Deviation from business requirements.
• This is not up to market standards and needs enhancements to match current market standards.

Success factors for This Testing

Once this test is planned, prepare a checklist that increases its success rate. There are some action items that are to be followed before the Acceptance test starts.

They are:

• Have a well-defined scope and make sure there is a business need for the scope identified for this testing.
• Execute Acceptance tests in the System testing phase itself at least once.
• Perform extensive ad-hoc testing for each of the acceptance test scenarios.

Conclusion

In a nutshell, Acceptance testing helps in figuring out the efficiency of development and testing teams.

There are several tools to conduct this activity, but usually, it is preferred to be done manually as there is an involvement of the real users and different stakeholders who are not from a technical background, and it may not be feasible for them.

What’s next?

In our next tutorial, we will hover over the following topics:

• Acceptance test criteria examples.
• How to write an Acceptance Test Plan.
• A suitable template for Acceptance Test writing.
• How to write Acceptance tests with examples.
• Identify Acceptance Test scenarios.
• Acceptance test reports.
• Acceptance testing in Agile and test-driven development.

NEXT Tutorial #2: Acceptance Test Plan

Have you performed Acceptance Testing? We would be glad to hear about your experiences!!