Applying Risk-Based Testing Concepts: RBT Case Study

This Tutorial Details the Theory & Application of the Concepts of Risk-Based Testing. The article Includes a Case Study to Explain Practical Application:

Development Delayed – Testing Time-Crunched & No Change in Target Date! Exhaustive Test Coverage Impossible

So, What do we do ??

As per the definition, Risk-Based Testing is a testing approach that prioritizes the tests of features and functions based on the risk of their failure and importance to the business – a function of their importance and likelihood or impact of failure.

Though Risk-based testing concepts are detailed in every other testing website, many of them claim to follow risk-based testing procedures. How much testing they perform is true, but “Risk-Based” is a question for thought.

Having said this, this tutorial details the concepts of Risk Based Testing, which when understood better, would help us to do our testing very efficiently. This tutorial presents us with a framework, that could be used to identify the test scenarios based on Risk Assessment.

It also involves a real-time Case study where Risk Based testing approach was followed religiously, by providing efficient and reliable test coverage and of course a satisfied customer.

Risk Based Testing: Overview

Recently, organizations are too bent towards producing high-quality deliverables in breakthrough speed with reduced cost. Of all these, there is always a delay in getting clarity of the requirements.

Testing is at the end of the software life cycle in waterfall/iterative methodologies, QA is being pushed to “finish testing” by finding more defects before it escapes to the customer. In the past few years, most software’s have strict time-to-market issues resulting in tight schedules imposed on project teams to deliver on time.

This leaves the project manager with different options on how to go about planning project activities and deliverables. Our Testing should be planned and designed, largely through identifying, prioritizing, and addressing potential risks.

How do we do all these with the available resources and time? Here is where Risk-based testing can be considered.

A structured risk based testing process supports the delivery of high-quality applications within the compressed timelines. In a way, it assures the quality of the product by meeting customer requirements. It participates measurably in project success by reducing the risk of misdirected and unfulfilled requirements.

Risk Based Testing Approach

According to Webster’s New World Dictionary, the risk is “the chance of injury, damage or loss; dangerous chance; a hazard”.

The risk-based testing approach focuses on finding the more important defects first. Everyone would accept this fact. But our main challenge lies in identifying the right areas where important defects are hidden.

Here, the Pareto principle which says “80% of the defects comes from 20% of the cases” could not be of much help. This is because if a business-critical requirement does not fall in that 20%, then this would mean that we are at the risk of providing low-quality products to the customer.

Now how do we ensure that our critical business requirements fall in that 20%? We would require a methodology that would identify functions in their system where the consequence of a defect would be most costly (either to the vendor or to the vendor’s customers) and also a technique to identify those functions with the highest probability of defects.

Risk analysis needs to be performed and the function with the highest risk exposure, in terms of probability and cost needs to be identified. A risk-based testing approach that requires the testing resources to be focused on those areas representing the highest risk exposure needs to be followed.

Risk-Based Testing: Phases

Risk analysis activity model. This model is taken from Karolak’s book “Software Engineering Risk Management”, 1996 [6] with some additions made (the oval boxes to identify the phases) to show how this activity model fits in with the test process.

Case Study

The Application

This case study is based on the system and integration test stages of a re-engineering project. The project scope includes re-engineering the legacy database hosted in DB2 to Oracle which encompassed in itself re-engineering of the data model, migrating the existing data to the new one, thereby demanding the required changes to interfaces.

During this project, large investments were made in terms of productivity tools, design, Quality assurance, and quality control.

The project was executed in 2008. The project consumed approximately 48 man-months and the testing constituted around 30% of the total project resources.

Project Description

This project involved the migration of the existing DB2 database to the Oracle Database. The main purpose of the project is to move to the latest oracle DB that had the features of fast response and ease of usability and scalability.

Functional Testing

The Functional Testing for the project involved 3 Stages as mentioned below:

• Database / System Testing
• Integration Testing
• Regression Testing

Testing had to be carried out for nearly 11 products across 6 countries to ensure that the data and functional integrity is not impacted after all the database and application changes have happened.

#1) Database/System Testing

Database/System testing involves all the below-listed test activities.

• Database migration testing from DB2 to Oracle
• Data validation and verification testing
• Stored Procedure Testing
• Batch Testing

#2) Integration Testing

Integration Testing was done from the application’s end to ensure that database migration changes and the stored procedure changes are closely coupled with the calling application.

End to end Testing from the UI to make sure that the data integrity is not impacted for any Stored Procedure Changes.

#3) Regression Testing

Regression Testing was performed once the system and integrations tests were completed to ensure that all the changes still hold good from the UI end.

The front end application which was previously fetching data from the DB2 application has to change its source database to the oracle database.

Non-Functional Testing

The Non-functional Testing was done for two main features:

(i) Database Performance Testing: Bulk data proceeded through different stored procedures and the response time was calculated to verify whether the response time meets the agreed SLA.

(ii) Migration of Bulk Data: A Bulk amount of data was to be migrated from the existing DB2 to Oracle database to calculate the total time required for the entire migration.

Why RBT?

(i) Time Constraint

Even though we had strict timelines, we had to do full-fledged functional testing and make sure that the customer journey and the integrity of the data are not impacted.

(ii) Technical Constraints

The test as well as development team members were new to database migration activities and had initial struggles to catch up with all the technical aspects of the project. There were millions of records to be migrated from the existing DB2 to Oracle.

Testing the entire set of the migrated records is impossible. QA team first identified the scenarios for which the migrated data needs to be tested. 5% of the entire migrated universe was taken under the test suit based on the various scenarios identified. This was a challenging task.

There were nearly 50 Stored Procedures which changed during the migration. The identification of test data for all these stored procedures was a challenging task.

Applying Risk-Based Testing

The Start

We had provided the test strategy first to the customer documenting the project scope, test approach, and test deliverables. Our test approach was supposed to be in a very traditional way with everything well documented before the test execution.

It was all very good except that it did not have any flexibility in finishing the testing on time because of the larger scope. We realized that it was very much impossible to test the whole application and database here by considering the vast project scope – 11 products across 6 countries.

The Analysis

This is the time when we started pondering over “minimum level of testing” or “just enough testing” including the level of documentation for all the functions and identify how a Risk Analysis would be used to identify the functions to be focused on during the test execution.

We realized that:

• We had to perform Risk Analysis to identify the most critical areas both to the customer as well as his business users.
• Structuring of Test Process had to be improved, or even “optimized”. This included test planning, redefining the test design approach, and also convincing the customer to prove the quality of the product being delivered.

The Approach

The approach we took here are detailed in the steps below:

• We, with the help of the customers and business analysts, identified those areas or requirements of the highest risk that will impact their businesses. (Risk Identification & Risk Analysis).
• After having acquired the list of risk-prone requirements, we had performed a risk analysis after which, we were able to prioritize the requirements. (Risk Assessment).
• Risk prioritization involved assigning a risk factor.
• Based on the risk prioritization, the testers were able to come up with test scenarios covering the high-risk areas. (Risk Strategy).
• Risk Strategy was well documented and after a couple of reviews, it got approved from the client’s end.
• During test execution, the identified scenarios were executed and reported accordingly (Risk Reporting).
• More focus was applied to the highest risk-prone areas (Risk Mitigation).

Risk Assessment And Analysis

To get the right test scenarios tested and also to prove the customer that we are selecting the right test scenarios, we had a detailed Risk Assessment and Analysis which is specified in the spreadsheet below.

The numbers shown are just samples. Weightage for the scenarios is something that businesses can decide and accordingly, the risk exposure will be calculated. Here 60% weightage has been provided for Usage Frequency and Visible areas and Business criticality 20% weightage has been provided.

Every requirement was analyzed thoroughly with the help of the business analysts and the business teams and a summary of the below was arrived at. This was after very careful reviews from the client.

Guidelines for Business Prioritization is mentioned in the below section – Appendix A.

Appendix A

The weightage values and the criteria provided below are just cited for example. The teams can define their prioritization guidelines.

Reporting

Reporting our test execution in the perspective of Risk-Based Testing could be done in many ways. Daily tracking in the above project was performed as below.

We displayed the project status in terms of Test Coverage & Defect Summary including the risk exposure as mentioned below.

Benefits

• Test effort is tightly calibrated to the level of risk reduction that any given functional or non-functional attribute requires.
• Tests are run in an order that maximizes the chances of discovering the most important bugs early in test execution.
• If necessary due to schedule pressures, less important tests (which are sequenced towards the end of the test execution) can be eliminated.

Challenges

As we worked towards applying Risk Based testing technique by analyzing its benefits, of course, some challenges came along. These challenges are not very specific to a project case but are common for all the projects.

1. Risk Prioritization is the most critical activity here and it should be performed carefully after analyzing all the criteria of business criticality. Persons who are well aware of the domain and business processes should be involved in this activity.
2. We should understand that there is always an underlying residual risk that occurs as a result of test cases not being executed because of low risk. This low risk should be consciously accepted by all the stakeholders.

Conclusion

Risk-based testing included with meticulous planning helped this project case to be successfully implemented across all 6 countries on time.

To summarize, Risk-based testing could prove successful only when the business teams join together during the Risk analysis and prioritization phase to understand and identify the highly impacted scenarios, otherwise, it could prove to be a disaster.

References – Risk-Based Testing