Risk Based Testing: Risk Management in Software Testing

This comprehensive guide provides valuable insights into risk-based testing, risk management, and its approach, illustrated with practical examples.

What is Risk Based Testing?

Risk-based testing is to carry out testing or to design and execute the scenarios, such that the top business risks which will hurt the business as identified by the customer are unearthed in their product or feature early in the life cycle and are mitigated by implementing mitigation measures.

=> Click Here For Complete Test Plan Tutorial Series

The negative impact can include cost impact, dissatisfied customers, bad user experience, and even losing customers.

The RBT approach is to ensure that testing is done in such a way that even if a user finds a bug in the production, that does not stop him/her from using the software or does not seriously impact the business.

RBT is testing carried out based on product risks. By employing a prioritization technique for test cases, RBT assesses the risk and impact of feature failure on the business in terms of cost and other damages.

Hence, risk-based testing uses the principle of prioritizing the tests of the features, modules, and functionalities of a product or software. The prioritization is based on the risk of the likelihood of failure of that feature or functionality in production and its impact on the customers.

Risk-Based Testing And Its Relevance To Agile & DevOps

Three hundred hours spent on developing software can be made useless in just 30 seconds with a single defect identified in production.

This can ruin the purpose of the whole product with no other option but to withdraw it from the market. And that is the significance and need for ‘Quality Testing’.

As technology advances, the software is now cloud-based, accommodating various operating systems, platforms, and complex IT infrastructures. Consequently, end-users have become increasingly demanding regarding features and options, never compromising on customer satisfaction.

Nowadays, ‘Quality’ is becoming a crucial factor in software delivery, where continuous improvements are enhancing the quality to keep the customers happy.

We often notice that almost all the testers commonly face tremendous pressure as their testing window is squeezed, and typically, the build is handed over to them for testing at the last minute. There is not enough time and resources for them to run all the tests that they have designed and the automation coverage is not always 100% and it has its challenges.

Meeting the delivery timeline and maintaining quality are equally important. Whatever Plan B, to add additional test resources by pulling out from the other teams, is not working out. Plan C, stop doing all the other activities and divert the effort to this alone, is not helping. However, much addition of test resources is not working out.

There is no other option but just to run limited and important tests within the available time and resources.

So, how do we decide which test is important at this stage? Whatever a tester considers important may not be important for the customers. From whose perspective is the importance of the feature or a functionality decided? Who will decide which are the important tests? And a lot of other questions keep arising.

To answer all these questions and to handle the above situation effectively, a testing approach called ‘Risk-Based Testing’, shortly called ‘RBT’, came into existence, where the team has planned and identified the test scenarios based on the ‘Project Risk’ criteria.

Though the QA team has a clear picture of the important tests, RBT is a proven method of identifying the crucial tests from the customer and business perspective through a ‘Risk Analysis’ procedure.

So, now as against the traditional way of simply identifying the defects in the software, the QA approach and goal have changed with time due to the change in the technology, increase in the competition in the market to release quality software, introduction of ‘Automate Everything’, and in totality introduction of Agile and DevOps practice of delivering the software over few hours.

As a result, the current ‘Testing Principle’ trend goes beyond simply identifying defects.

Risk-Based Testing Approach

It is always like preparing for an examination, that one cannot say testing is enough and there are no more defects in the software, even if they design and execute an ample number of tests.

Increasing the number of test cases alone won’t guarantee software stability indefinitely. In the present moment, the focus should be on meeting the customer’s expectations rather than solely on the number of tests.

Hence, it is essential to strike a balance in optimizing the testing to get the maximum benefit from the reasonable effort of testing. And this is more important when the testing timelines are very tight and enough resources are not available to carry out sufficient testing.

Thus, in this case, the RBT approach plays a key role in optimizing the QA effort and maximizing the testing benefit with the minimum business risk.

So, if we focus on the above aspect, then the work of a QA will be much better. They do not have to burn their weekends in the office, continuously testing the software and getting worried about every S4 (Severity 4) and P4 (Priority 4) defect that comes out of the testing.

Well, 4 is considered the lowest priority and severity of the defects in testing. It would be more advantageous for them to allocate their time to other useful areas of the project.

To summarize the key drivers of the ‘risk-based testing’ approach:

• To enable testing ‘what customers want’ from a business perspective.
• To meet the time schedule with expected quality.
• To optimize the QA effort.

When Do We Use The RBT Approach

This is used under the below scenarios:

• RBT approach can be used whenever there is a limitation or constraint on the time, cost, and resources of a project and whenever there is a need to optimize the resources.
• RBT approach is used when the program is more complex and adapts new technology and hence involves a lot of challenges.
• When the program is an R&D project and it is of a first-kind type and there are several unknowns and risks in the project.

RBT Approach Example

Several risk-based analysis approaches are used in the IT industry to overcome the risks faced by production and its impact.

Given below is one such approach:

This approach of RBT includes identifying the ‘vital functionalities or key features’ of the product assessing the risks to which each of these functionalities gets exposed to in the production and implementing appropriate mitigation measures in place to lower or mitigate the risk.

Hence, the RBT approach includes testing the functionalities which have the probability of failure and the highest impact on business. The failures could be operational or business, technical, external, etc.

Ways To Carry Out Risk Analysis

There is no standard process or template defined as such to carry out the risk analysis in software testing for every feature of a product. Various organizations use their own approach to risk analysis methods.

Risk analysis can be carried out on various project items to identify the risks and to implement an RBT approach for analysis. Those items include:

• Features
• Functionalities
• User Stories
• Requirements
• Use Cases
• Test Cases

Here, let us focus only on test cases to understand the Risk-Based Testing approach implementation.

Risk Analysis Procedure

Risk analysis includes the involvement of relevant stakeholders of the program from both the ‘Technical Team and Business Team’, which includes owner of the product, Product Managers, Business Analysts, Architects, Testers, and Customer Representatives.

To identify the importance of each feature of a product and prioritize them based on the risk of failure and its impact on the end-users in production, we would organize brainstorming sessions involving these stakeholders to carry out the discussion.

Various ‘project documents such as Requirements Document, Technical Specification Documents, Architecture and Design Documents, Business Process Document, Use Case Document, etc., will become the input for the brainstorming session.

Stakeholder’s knowledge about the product and the existing product in the market will also be an input factor for the discussion.

Few other sources of inputs can also include:

• To gather inputs on the most used functionality.
• Inputs from consulting a domain expert.
• Data from the previous version of the product or a similar product in the market.

During the brainstorming session, the risks pertaining to each of these features are identified. The types of risks could be an operation, technical or business-related. The tests and scenarios related to them are weighted and risk values are assessed based on the likelihood of risk occurrence and impact of the risk.

The ‘likelihood of risk occurrence’ can be due to:

• The product development team lacks a proper understanding of the feature.
• Improper architecture and design.
• Insufficient time to design.
• Incompetency of the team.
• Inadequate resources – people, tools, and technology.

If failure occurs, the ‘impact of the risk’ refers to the consequences for users and the business. The potential consequences could be:

• Cost impact, resulting in a loss.
• Business impact resulting in losing business or losing market share, law proceedings, loss of license.
• Quality impact, resulting in the substandard or incompetent product release.
• Bad user experience, resulting in dissatisfied and loss of a customer.

The focus area of assessing the risk of a feature or product can be,

• Area of business criticality of the functionality.
• Most used features and important functionality.
• Defect prone areas
• Functionality bearing the safety and security impact.
• Area of complex design and architecture.
• Changes made from previous versions.

Risk Analysis Methodology

Let us understand the ‘risk-based testing methodology’ now.

The risk-based testing approach uses RISK as the criteria at all test phases of the testing cycle, i.e., test planning, test design, test implementation, test execution, and test reporting. Ideally, one can design a numerous number of test scenario combinations.

Hence, the RBT approach includes a ranking of the tests based on the severity of the risks to find out the most defective or risky area of failure, which causes a high impact on the business.

The main objective of risk analysis is to distinguish between the ‘High Value’ items like product features, functionalities, requirements, user stories, and test cases, and ‘Low Value’ ones and hence later to more focus on ‘High Value’ test cases, by less focusing on ‘Low Value’ test cases. This is the initial step of risk analysis before starting the risk-based testing.

The main task of categorization or grouping of test cases into High Value & Low Value and assigning the priority value to each of these test cases includes the following steps:

Step #1: Using a 3X3 grid

During risk analysis, a team of stakeholders evaluates the ‘Likelihood of failure’ and ‘Impact of failure’ for each functionality, non-functionality, and its related test cases using a 3X3 grid.

The likelihood of failure of each functionality in the production is accessed by a group of ‘Technical Experts’ and are categorized as ‘Likely to fail, likely and unlikely’ along the vertical axis of the grid.

Similarly, the ‘Impact of failure’ of these features and functionalities in production is experienced by the end customer. ‘Business Specialists’ evaluate untested items and placed in the ‘Minor, Visible, and Interruption’ categories along the horizontal axis of the grid.

Step #2: Likelihood and Impact of Failure

All the test cases are positioned in the quadrants of the 3 X 3 grid based on the identified values of a likelihood of failure and impact of failure, which are shown as dots in the picture below.

Obviously high likelihood of failure and high impact of failure (interruption) are grouped in the top right corner of the grid, which is of high importance and hence it is identified that ‘High Value’ tests and ‘Low Value’ tests are grouped in the bottom left corner which is of least or no importance to the customer, where minor focus can be given to these features or test cases.

Step #3: Testing Priority Grid

Based on the above positioning of the test cases in the grid, we prioritize and label the tests with priorities 1, 2, 3, 4, and 5, and mark them against each of them. The most important tests are positioned in a 1^st grid that is assigned with priority 1 and similarly less important ones are ranked as 2, 3, 4 and 5.

Finally, all the test cases are sorted based on their priority numbers and are picked up for execution in the order of priority. The system executes the test cases with high priority first, while either executing the ones with low priority later or removing them.

Step #4: Details of Testing

The next step is to decide on the level of details of testing for the defined scope of testing. Based on the above ranking, we can decide the depth of the scope of the testing using the below grid.

High priority tests with ranking 1 are more thoroughly tested and, experts are deployed to test these high criticality features and their related test cases. Similar is the case for test cases with priority 2, 3, and 4. A decision to de-scope priority 5 features and tests based on the available time and resources can be taken.

Hence, Level of Detail of Testing approach of prioritizing the features and its test cases not only helps the Testers to identify the ‘High-Value Tests’ but also guides them to decide on their ‘detail level of testing’ based on these priority rankings and helps them to carry out better testing and reduces testing cost by optimization process.

How is RBT Relevant to Agile and DevOps

Now, after understanding the Risk-Based Testing approach of carrying out the testing based on the prioritization of tests depending upon the ‘Risk of Failure’ of a particular feature and its ‘Impact to Customer’ in live, obviously one would raise the question of the relevance of Risk-Based Testing approach in Agile and DevOps Practices.

‘Automate Everything’, ‘Test Everything’, ‘Test Continuously’, ‘Test Repeatedly’ are the key concepts of these practices.

Each time, when there is a change in the code or there is a release, all the designed tests are run through the automated Continuous Integration(CI) / Continuous Delivery(CD) pipeline quickly and repeatedly, irrespective of their prioritization.

So, how are RBT and DevOps related? Where does RBT fit in and become applicable in the context of Agile and DevOps?

#1) I previously mentioned that not all industries and products have 100% automation coverage for testing. So, if at all the team has to make the choice of prioritization for test execution from the choice of manual test cases and would like to spare the energy and effort of the test resources towards other activities, then RBT is the best choice.

The risk-based approach is also a better bet for running automated tests with high priority ones and testing at the earliest.

#2) QA team can adopt the RBT approach more effectively during the Requirement Analysis in analyzing the requirements and providing an advance report of the probable risks of the products and the features so that appropriate actions can be taken proactively by the program team to mitigate it.

#3) By using the RBT approach, test cases and scenarios can emphasize high-risk features and functionalities.

#4) Identifying the ‘High Risk’ areas enables the QA team to focus their testing effort on those areas to test more thoroughly using highly skilled testers.

#5) ‘Fail Fast’, as we all know is the concept of ‘Agile’ and ‘DevOps’, for which RBT approach helps to identify the ‘high risk’ areas in the software, identify the defects early and allow them to fail fast and fail first and let the team fix it.

#6) The goal of Agile/DevOps is customer focus and hence the RBT approach enables the QA to focus on customer experience than just finding defects.

Benefits of Risk-Based Testing approach

We already understood the purpose and use of the RBT approach of analyzing the requirements, designing and executing the testing scenarios. There are several benefits of RBT.

The benefits of Risk-Based Testing can be summarized and organized as:

• Helps in a more efficient and optimized use of test resources.
• Helps in easing the QA work, testing, test design & development, and test preparation activities by prioritizing.
• Helps in better managing the QA resources by allocating the key resources towards high focus areas.
• It helps in the effective utilization of resources and diverts their time and energy on better things in the project.
• Helps the QA team in planning the testing efforts based on the risk assessment and identification of volatile and high-risk areas.
• It helps the team to optimize the tests to be conducted depending on the importance and hence de-scope low-value tests in agreement with the stakeholders.
• Overall, it helps in cost reduction through optimized and reduced testing activities.
• RBT approach enables the QA team to test the high-risk areas first and allows the product to fail fast and fix fast.
• RBT approach helps in bringing clarity to the ‘Test Coverage’ and the ‘Test Scope’ of the entire group of stakeholders.
• The team can increase their focus on high-risk areas and keep less focus on the low-risk area.
• RBT allows the team to decide well in advance on implementing the most effective way of mitigation for the product risks.
• RBT helps in avoiding the effect of the inappropriate implementation of mitigations.
• Risk-based testing allows the team to take action either to mitigate or to plan for contingency or to position any workaround to overcome the failure or reduce its impact if the risk occurs live.
• RBT helps in minimizing the residual risk in the release.
• It helps in achieving improved quality through less costly defects in production.
• Ultimately helps in improved customer experience and satisfied customers.

Next, we will learn how to manage risks at Test Planning and Test Execution phases of the Software Testing Life Cycle.

---

Risk Management During Test Planning

How to manage risks during the test planning phase:

Life is full of risks, and so is a software project. Anything can go wrong at any time. We are always on our toes to make things right–but what about making sure that nothing goes wrong and that when it does we know exactly what to do? Enter risk management – this is a portion of a software testing project that prepares us to prevent, understand, find, and get over risks.

A risk is simply a problem that is likely to occur and when it occurs, it will cause a loss.

The loss could be anything- money, time, effort, or a compromise in quality. The loss is never good. No matter how much of a spin we give it, it’s not positive and never will be. Therefore, risk management is an integral part of software projects to make sure we handle risks and prevent/reduce losses.

Risk-based testing: Since we are a QA community, let us stay specific to risks and the process related to it in our QA world exclusively. Risks are assessed and managed roughly in 2 phases of our Software Test life cycle. STLC can be categorized into 3 phases – Test Planning, Test Designing, and Test Execution.

The risk management process occurs twice, during:

1. Test Planning
2. Test Case Design(end) or sometimes in the Test Execution phase

It is mandatory in case 1 but with case 2 it is more of a ‘need-basis’ situation.

Two-part article series:

Even though the underlying process is the same, the types of risks addressed in both these areas are completely different. In order to do justice to them individually, we are going to handle them differently as a two-part series.

This section is going to be about “Risk Management during Test Planning”.

Risk Management Process

The generic process for Risk Management involves 3 important stages:

1. Risk Identification
2. Risk Impact Analysis
3. Risk Mitigation

Testing Risks and Mitigation Examples:

#1) Risk Identification

As it is said, the first step to solving a problem is identifying it. This stage involves making a list of everything that might come up and disrupt the normal flow of events.

The main outcome of this step is a list of risks.

The QA lead/Manager/representative commonly led to this risk-based testing step. However, the lead alone cannot come up with the entire list – the entire QA team’s input makes a huge impact. We can say this is a collective activity led by the QA lead.

Also, the risks that are identified during the Test planning phase are more ‘managerial’ in orientation- meaning we are going to look at anything that might impact the QA project’s schedule, effort, budget, infrastructure changes, etc. The focus here is not the AUT, but the way the QA phase will go on.

Risks During Test Planning: Risk-Based Testing Examples

The following is a sample list of risks that might be listed during the Test Planning phase. Please note that the AUT and its functionality are not the focus here.

1. The testing schedule is tight. The test cannot be prolonged beyond the UAT scheduled start date if the beginning of testing is delayed due to design tasks.
2. Not enough resources, resources onboarding too late (the process takes around 15 days.)
3. Defects are found at a late stage of the cycle or at a late-cycle; defects discovered late are most likely to be because of unclear specifications and are time-consuming to resolve.
4. The scope is not defined completely.
5. Natural disasters
6. Non-availability of Independent Test Environment and accessibility
7. Delayed testing because of new issues.

You can choose to be as thorough as you would like, depending on the amount of time available.

Once all the risks are listed, we progress to risk assessment/risk impact analysis.

#2) Risk Assessment/Risk Impact Analysis

Is your Risk Analysis something like this? 🙂

Risk Analysis in Software Testing: All the risks are quantified and prioritized in this step. Every risk’s probability (the chance of occurrence) and impact (the amount of loss that it would cause when this risk materializes) are determined systematically.

High – medium – low values are assigned to both the probability and impact of each risk. The risks with “high” probability and “high” impact are taken care of first and then the order follows.

Risk impact analysis table:

Following these steps, the risk impact analysis table for the above risks listed would look something like this (all values are hypothetical and only for understanding purposes):

#3) Risk Mitigation

The final step in this Risk-Based Testing (RBT) process is to find solutions to plan how to handle each one of these situations. These plans can differ from company to company, project to project and even person to person.

Risk Mitigation Techniques:

Here is an example of what the risk table transforms into when this phase is complete:

A few points to note:

1. The sooner risk management starts in a QA project planning phase, the better.
2. Of all 3 steps, risk identification is the most important. If anything is not listed and considered for further steps, the risk goes unhandled.
3. Try to find an ideal time frame for this activity. Remember, too much planning leaves too little time for doing.
4. Also, after the risk management process, if a new situation comes up, the risk management plan can be altered or updated to reflect the most current condition.
5. Historical data can be very useful for the success of this process.

This brings us to the end of risk management in the test planning phase. Even though the underlying steps and principles are similar, the risk management process is more concentrated towards the AUT when it happens in the test designing/execution phase.

In our next section, we will cover–Risk Management in the Test Execution phase.

---

Risk Management At Test Execution Phase (with Example)

In our journey to understanding the risk management process, we talked about how it goes exclusively in the test planning phase of risk-based testing. We also understood the generic process that involves – Risk identification, Risk Assessment, and Risk Mitigation Plan.

How to manage risk at the test designing or test execution phase:

There is one other form of Risk Management (also sometimes referred to as, Risk-based testing) that occurs during the Test designing or Test execution phase depending on the situation. Now, what situation are we talking about? Let us try to understand that first.

We all know that our testing work is reactive. No requirements (or scope defined), we can’t perform a feasibility analysis and write test scenarios or plan for testing activities.

Similarly, when the code is not ready, we have nothing to test, no matter how much of prep-work we might have been ready within terms of test cases, test data, etc. Also, testing is the only step left before the product goes live.

Risk Management – With a Focus on The AUT

Let us understand this better with an example:

When testing starts on Jan 1st and lasts until Jan 14th, the Go-live date for the product is typically set immediately. Let’s say- Jan 15^th for simplicity’s sake. Now, in a perfect world, things would go exactly as planned. But we all know the reality.

Here, let us assume that the testing did not start until Jan 7^th, which means we lost half of our testing time. But we need 14 days to test the product thoroughly. We could move the go-live date farther by 7 days- however; this usually is not an option. Because the product is promised to hit the market on a certain date and any delays are not good for the business.

Testing teams often face delays and must find ways to compensate while still ensuring thorough testing of the product. This job is challenging, isn’t it?

This is where the Risk Management process is again employed.

• Now, if delays are anticipated ahead of time before even the testing begins, the process takes place in the test designing phase.
• If delays happen during a Test Execution phase that started normally, the process is followed during the test execution phase.
• The steps and the method are the same no matter what stage it happens.

What is the process?

Risk management takes place to determine what areas of the AUT (Application Under Test) need maximum focus. These are typically the functional areas (modules or components) that are critical for the success of the final product and are most prone to failure.

Read also => Failure Mode And Effects Analysis (FMEA) is a Risk Management technique

Who performs it?

Since it is regarding the AUT, the knowledge about it is not only with the QA but with all the other teams – Dev, BA, Client, Project management teams, etc. Therefore, it is a collective effort, driven by the testing team.

How Does Risk Bases Testing Take Place

Step #1: Risk Identification

Identify all the functional areas of the AUT. This will simply include making a list.

Step #2: Risk Assessment

All the risks are quantified and prioritized in this step. Quantifying is simply assigning a number to each risk in the list that will give an indication of the priority with which it needs to be addressed.

Every risk’s probability (the chance of occurrence) and impact (amount of loss that it would cause when this risk materializes) are decided.

The typical method is to assign the ratings. For example, probability can take values 1 to 5. 1 being the probability of occurrence being low (not likely to happen at all) and 5 being high (will most certainly happen).

Similarly, the impact can also be assigned a 1-5 rating. 1 being low impact (even if this risk materializes, the loss is minimal) and 5 being high impact (huge losses when happens).

Step #3:

Make a table format and circulate to all the team reps- Dev, BA, Client, PM, QA and anyone else relevant.

Step #4:

Instruct each team to fill in the values based on their rating for probability and impact. Since the probability and impact values are numeric, it will make the calculating of the “Risk Factor” value easy.

Risk factor = Probability X Impact. The higher the risk factor, the serious the problem is.

An example:

Please note that at this point, this is simply the outcome of one team’s rating. For a project involving 5 different teams, the QA team would end up with 5 different tables.

Step #5:

Take an average of the Risk factor values. For example, if there are 5 teams, for each module, add all the risk factor values and divide it by 5. These are the final values we are going to deal with. Say these are the averaged risk factors:

The more the risk factor, the more that module has to be tested.

So, Module 5 and 2 are most crucial for the success of the business. Share the results with all of the teams.

Step #6:

Risk Mitigation Plan: Now, this is the step that changes from Project to Project. We have identified that modules 5 and 2 are the ones that need to be concentrated on most.

Examples of the plan could be:

1. Our thorough testing of modules 5 and 2 will involve testing all the test cases related to them. The other modules will be tested in exploratory.
2. Modules 5 and 2 are going to be tested first and then, depending on the time available, the others are going to be taken care of.

Once a plan is made, all the teams agree and follow it to test the product, keeping into account the risk-factor.

That’s it!

A few important points to note:

• Since this is a collective activity that takes everyone’s opinion into consideration; the chances of it being accurate and effective are higher.
• This is not a formal method and does not have to be a part of every QA project.
• Sometimes, even if the team chooses not to draw tables and assign values, a simple brainstorming session with everyone present can give the QA team a good direction on how to proceed.
• The development team’s input is very important because they are the ones who create the product, so they will know what might work and what might need additional checking. Be sure to be on the lookout for that.
• Even though there are multiple steps, it does not take a considerable amount of time to perform them. Especially if all the teams can sit together and work simultaneously.
• Remember, this process and its outcome is only the alternative. Getting as much time as planned for testing is the best way to perform the QA activity.

Conclusion

The risk-based testing approach indicates that the focus of the tester is not just to keep exploring the defects, irrespective of severity and priority. Now things have changed and Testers have to work smart and they need to understand the clear ‘Need of the Customer and Wants of the User’.

They need to study the product thoroughly and understand which is the most frequently used feature in the production, which is the most critical path for revenue generation and how to protect and safeguard the customers from the production issues and business threats.

Hence, the RBT approach clearly educates the3 testers that just testing everything or testing extensively does not mean that testing is complete or there are no defects in the product. Testing effectively in a stipulated time and ensuring that critical and major business impacts are nullified and that is quite important for the tester.

Thus, risk-based testing is the most efficient tool for the QA team to guide the project stakeholders based on the project risks. RBT approach helps the QA team in the continuous identification of risk and its resolution that could endanger the achievement of the overall project goals and objectives and helps in achieving the ultimate aim of the QA Group.

P.S. The words QA and Testing have been used interchangeably throughout the document.

About the Author: Gayathri Subrahmanyam and Swati S. from the STH team are the authors of this article.

Gayathri is a Software Test SME with 2+ decades of experience in Software Testing and has extensively adopted the ‘Risk-Based Testing’ approach as a part of Test Industrialization in several engagements and has realized the benefit of Test resource optimization and quality testing. 

Was Risk Based testing a tough challenge for you? Are there any intriguing facts you can contribute to our tutorial? Share your thoughts freely in the comments below!

=> Visit Here For Complete Test Plan Tutorial Series