The Ultimate Guide To Risk Based Testing, Risk Management, and Its Approach with Examples:
What is Risk Based Testing?
Risk based testing is to carry out testing or to design and execute the scenarios, such that the top business risks which will have a negative impact on the business as identified by the customer are unearthed in their product or feature early in the life cycle and are mitigated by implementing mitigation measures.
The negative impact can include cost impact, dissatisfied customer, bad user experience, and even to the extent of losing the customers.
In other words, RBT approach is to ensure that, testing is done in a such a way that even if a user finds a bug in the production, that does not stop him/her from using the software or does not impact the business in a serious way.
RBT is testing carried out based on the product risks. RBT is to find out well in advance, as what is the risk of the failure of a particular feature or functionality in the production and its impact on the business in terms of cost and other damages by using a prioritization technique for the test cases.
Hence, Risk based testing uses the principle of prioritizing the tests of the features, modules, and functionalities of a product or software. The prioritization is based on the risk of likelihood of failure of that feature or functionality in production and its impact on the customers.
What You Will Learn:
Three hundred hours spent on developing a software can be made useless in just 30 seconds with a single defect identified in production.
This, in turn, can ruin the purpose of the whole product with no other option but just to withdraw it from the market. And that is the significance and need for ‘Quality testing’.
With the rapid growth in technology, software is hosted on the cloud supporting multiple OS, multiple platforms, complex IT infrastructures, etc., the end users are becoming more and more fussy about the features, options and they never compromise for customer satisfaction.
Nowadays, ‘Quality’ is becoming a crucial factor in software delivery, where continuous improvements are happening to improve the quality in order to keep the customers happy.
We often notice that it is a common problem for almost all the Testers to be under tremendous pressure that their testing window is squeezed and typically the build is handed over to them for testing at the last minute. There is no enough time and resources for them to run all the tests that they have designed and the automation coverage is not always 100% and it has its own challenges.
The delivery timeline cannot be missed and at the same time quality cannot be compromised as well. Whatever the plan B, to add additional test resources by pulling out from the other teams, is not working out, Plan C, stop doing all the other activities and divert the effort to this alone, is not really helping. However much addition of test resources, at the end, is not working out.
There is no other option but just to run limited and important tests within the available time and resources.
So, how do we decide which test is important at this stage? Whatever a Tester considers important may not be really important for the customers. From whose perspective is the importance of the feature or a functionality decided? Who will decide which are the important tests? And a lot of other questions keep arising.
In order to answer all these questions and to handle the above situation effectively, a testing approach called ‘Risk Based Testing’, shortly called ‘RBT’, came into existence, where the team has clearly planned and identified the test scenarios based on the ‘Project Risk’ criteria.
Though the QA team has a clear picture of the important tests, RBT is a proven method of identifying the crucial and important tests from the customer and business perspective through a ‘Risk Analysis’ procedure.
So, now as against the traditional way of simply identifying the defects in the software, the QA approach and goal has changed with the time due to the change in the technology, increase in the competition in the market to release a quality software, introduction of ‘Automate everything’, and in totality introduction of Agile and DevOps practice of delivering the software over a period of few hours.
Hence, the current trend of ‘Testing Principle’ is not just merely ‘identifying the defects’ but to also,
#1) Focus on the area of the product where there is a high impact on business due to failure or high likelihood of failure in the production.
#2) Focusing on identifying the defects early and allowing a team to fix it as early as possible and hence allowing the software/product or feature to ‘Fail Fast’.
#3) The most important aspect of Service of the QA team now is to focus on the customer in bringing value to the customer by increasing the focus on ‘end to end customer experience’.
It is always like preparing for an examination, that one cannot say testing is enough and there are no more defects in the software, even if they design and execute an ample number of tests.
There is a point at which software stability is not going to be doubly assured by increasing the number of test cases alone. At this point of time, it is not just to focus upon the number of tests but on what actually the customer is expecting from the release.
Hence, it is essential to strike a balance in optimizing the testing in order to get the maximum benefit with the reasonable effort of testing. And this is more important when the testing timelines are very tight and enough resources are not available to carry out sufficient testing.
Thus, in this case, RBT approach plays a key role in optimizing the QA effort and maximizing the testing benefit with the minimum business risk.
So, if we focus on the above aspect, then the work of a QA will be much relieved. They do not have to burn their weekends in the office, continuously testing the software and getting worried about every S4 (Severity 4) and P4 (Priority 4) defects that come out of the testing.
Well, 4 is considered as lowest priority and severity of the defects in testing. They can better invest their time in other useful aspects of the project.
To summarize the key drivers of the ‘Risk-based testing’ approach:
#1) To enable testing ‘what customers want’ from a business perspective.
#2) To meet the time schedule with expected quality.
#3) To optimize the QA effort.
This is used under the below scenarios:
Several risk based analysis approaches are used in the IT industry to overcome the risks faced by the production and its impact.
Given below is one such approach:
This approach of RBT includes identifying the ‘Vital Functionalities or Key Features’ of the product and assessing the risks to which each of these functionalities get exposed to in the production and implementing appropriate mitigation measures in place to lower or mitigate the risk.
Hence the RBT approach includes testing the functionalities which have the probability of failure and highest impact on business. The types of failures could be operational or business, technical, external etc.
There is no standard process or template defined as such to carry out the risk analysis in Software testing for each and every feature of a product. Various organizations use their own approach of risk analysis methods.
Risk analysis can be carried out on various project items to identify the risks and to implement RBT approach for analysis. Those items include,
In this case, let us focus only on Test cases to understand the Risk Based Testing approach implementation.
Risk analysis includes the involvement of relevant stakeholders of the program from both the ‘Technical Team and Business Team’, which includes, Owner of the Product, Product Managers, Business Analysts, Architects, Testers, and Customer Representatives.
Brainstorming session involving these stakeholders would be organized to carry out the discussion to identify the importance of each feature of a product and prioritize them based on the risk of failure and its impact to the end users in production.
Various ‘Project Documents’ such as Requirements Document, Technical Specification Documents, Architecture and Design Documents, Business Process Document, Use Case Document etc., will become the input for the brainstorming session.
Stakeholder’s knowledge about the product and the existing product in the market will also be an input factor for the discussion.
Few other sources of inputs can also include,
During the brainstorming session, the risks pertaining to each of these features are identified. The types of risks could be an operation, technical or business related. The tests and scenarios related to them are weighted and risk values are assessed based on the likelihood of risk occurrence and impact of the risk.
The ‘Likelihood of risk occurrence’ can be due to:
The ‘Impact of the risk’ is the effect of failure to the users and business if it occurs. The impact could be,
The focus area of assessing the risk of a feature or product can be,
Let us understand the ‘Risk based testing methodology’ in detail now.
The risk based testing approach uses RISK as the criteria at all test phases of the testing cycle, i.e., test planning, test design, test implementation, test execution and test reporting. Ideally, one can design a numerous number of possible test scenario combinations.
Hence, the RBT approach includes a ranking of the tests based on the risks severity to find out the most defective or risky area of failure, which causes high impact to the business.
The main objective of Risk analysis is to distinguish between the ‘High value’ items like product features, functionalities, requirements, user stories, and test cases, and ‘Low Value’ ones and hence later to more focus on ‘High value’ Test Cases, by less focusing on ‘Low value’ Test Cases. This is the initial step of Risk analysis before starting the risk based testing.
The main task of Categorization or grouping of Test Cases into High Value & Low Value and assigning the priority value to each of these test cases includes the following steps:
Risk analysis is performed using a 3X3 grid, where each functionality, non-functionality and its related Test cases are assessed by a team of stakeholders for its ‘Likelihood of failure’ and ‘Impact of failure’.
Likelihood of failure of each functionality in the production is generally accessed by a group of ‘Technical Experts’ and are categorized as ‘Likely to fail, quite likely and unlikely’ along the vertical axis of the grid.
Similarly, the ‘Impact of failure’ of these features and functionalities in production is experienced by the end customer, if not tested is assessed by a group of ‘Business Specialists” and are categorized under ‘Minor, Visible and Interruption’ categories along the horizontal axis of the grid.
All the Test cases are positioned in the quadrants of the 3 X 3 grid based on the identified values of a likelihood of failure and impact of failure which are shown as dots in the picture below.
Obviously high likelihood of failure and high impact of failure (interruption) are grouped in the top right corner of the grid, which is of high importance and hence it is identified that ‘High Value’ tests and ‘Low Value’ tests are grouped in the bottom left corner which is of least or no importance to the customer, where minor focus can be given to these features or test cases.
Based on the above positioning of the test cases in the grid, the tests are prioritized and labeled with priorities 1,2,3,4 and 5 and are marked against each of them. The most important tests are positioned in a 1st grid are assigned with priority 1 and similarly less important ones are ranked as 2, 3, 4 and 5.
Finally, all the test cases are sorted based on their priority numbers and are picked up for execution in the order of priority. The high priority ones are picked up for execution first and low priority ones are either executed later or de-scoped.
The next step is to decide on the level of details of testing for the defined scope of testing. The depth of scope of the testing can be decided based on the above ranking as per the below grid.
High priority tests with ranking 1 are ‘More Thoroughly’ tested and accordingly, experts are deployed to test this high criticality features and its related Test Cases. Similarly test cases with priority 2, 3, and 4. A decision to de-scope priority 5 features and tests based on the available time and resources can be taken.
Hence, Level of Detail of Testing approach of prioritizing the features and its test cases not only helps the Testers to identify the ‘High Value Tests’ but also guides them to decide on their ‘detail level of testing’ based on these priority rankings and helps them to carry out better testing and reduces testing cost by optimization process.
Now, after understanding the Risk Based Testing approach of carrying out the testing based on the prioritization of tests depending upon the ‘Risk of Failure’ of a particular feature and its ‘Impact to Customer’ in live, obviously one would raise the question of the relevance of Risk Based Testing approach in Agile and DevOps Practices.
‘Automate Everything’, ‘Test Everything’, ‘Test Continuously’, ‘Test Repeatedly’ are the key concepts of these practices.
Each time, when there is a change in the code or there is a release, all the designed tests are run through the automated Continous integration(CI) / Continuous delivery(CD) pipeline quickly and repeatedly, irrespective of their prioritization.
Then what is the relation between RBT and DevOps? Where would RBT fit in and become relevant in Agile and DevOps???
#1) Yes, as I said earlier, it is not that every Industry and every product has 100% automation coverage for their test executions. So, if at all the team has to make the choice of prioritization for test execution from the choice of manual test cases and would like to spare the energy and effort of the test resources towards other activities, then RBT is the best choice.
Risk based approach is also a better bet for running automated tests with high priority ones and testing at the earliest.
#2) QA team can adopt RBT approach more effectively during the Requirement Analysis in analyzing the requirements and providing an advance report of the probable risks of the products and the features so that appropriate actions could be taken proactively by the program team to mitigate it.
#3) RBT approach can be effectively used in designing the Test Cases and scenarios based on High risk so that more focus can be paid towards high-risk features and functionalities.
#4) Identifying the ‘High Risk’ areas enables the QA team to focus their Testing effort on those areas to test ‘More thoroughly’ using ‘High Skilled Testers’.
#5) ‘Fail Fast’, as we all know is the concept of ‘Agile’ and ‘DevOps’, for which RBT approach helps to identify the ‘high risk’ areas in the software, identify the defects early and allow them to fail fast and fail first and let the team fix it.
#6) The ultimate goal of Agile/DevOps is ‘Customer focus’ and hence RBT approach enables the QA to focus on Customer experience than just finding Defects.
We already understood the purpose and use of RBT approach of analyzing the requirements, designing and executing the testing scenarios. There are several benefits of RBT.
We can consolidate and list the benefits of Risk Based Testing as:
Next, we will learn how to manage risks at Test Planning and Test execution phases of Software Testing Life Cycle.
How to Manage Risks During Test Planning Phase:
Life is full of risks, and so is a software project. Anything can go wrong anytime. We are always on our toes to make things right – but what about making sure that nothing goes wrong and that when it does we know exactly what to do? Enter Risk management – this is a portion of a software testing project that prepares us to prevent, understand, find and get over risks.
A risk is simply a problem that is likely to occur and when it does occur, will cause a loss.
The loss could be anything- money, time, effort or a compromise in quality. The loss is never good. No matter how much of a spin we give it,it’s not positive- and never will be. Therefore Risk management is an integral part of software projects to make sure we handle risks and prevent/reduce losses.
Risk-Based Testing: Since we are a QA community let us stay specific to risks and the process related to it in our QA world exclusively. Risks are assessed and managed roughly in 2 phases of our Software Test life cycle. STLC can be categorized into 3 phases -Test planning, Test designing and Test execution.
The risk management process occurs twice, during:
It is mandatory in case 1 but with case 2 it is more of a ‘need-basis’ situation.
Two part article series:
Even though the underlying process is the same, the types of risks addressed in both these areas are completely different. In order to do justice to them individually, we are going to handle them differently as a two-part series.
This section is going to be about “Risk Management during test planning”.
The generic process for Risk management involves 3 important stages:
Testing Risks and Mitigation Examples:
As it is said, the first step to solving a problem is identifying it. This stage involves making a list of everything that might potentially come up and disrupt the normal flow of events.
The main outcome of this step is a list of risks.
This risk-based testing step is commonly led by the QA lead/Manager/representative. However, the lead alone will not be able to come up the entire list- the entire QA team’s input makes a huge impact. We can say this is a collective activity led by the QA lead.
Also, the risks that are identified during the Test planning phase are more ‘managerial’ in orientation- meaning, we are going to look at anything that might impact the QA project’s schedule, effort, budget, infrastructure changes, etc. The focus here is not the AUT, but the way the QA phase will go on.
Risks During Test Planning: Risk Based Testing Examples
The following is a sample list of risks that might be listed during test planning phase. Please note, that the AUT and its functionality is not the focus here.
#1. The testing schedule is tight. If the start of the testing is delayed due to design tasks, the test cannot be extended beyond the UAT scheduled start date.
#2. Not enough resources, resources onboarding too late (process takes around 15 days.)
#3. Defects are found at a late stage of the cycle or at a late cycle; defects discovered late are most likely be due to unclear specifications and are time-consuming to resolve.
#4. Scope not defined completely defined
#5. Natural disasters
#6. Non-availability of Independent Test environment and accessibility
#7. Delayed Testing Due To new Issues
At this point, you can choose to be as thorough as you would like, depending on the amount of time available.
Once, all the risks are listed, we progress to Risk assessment/Risk impact analysis.
Is your Risk Analysis something like this? :)
Risk Analysis in software testing: All the risks are quantified and prioritized in this step. Every risk’s probability (the chance of occurrence) and impact (amount of loss that it would cause when this risk materializes) are determined systematically.
High – medium – low, values are assigned to both the probability and impact of each risk. The risks with “high” probability and “High” impact are taken care of first and then the order follows.
Risk impact analysis table:
Following these steps, the risk impact analysis table for the above risks listed would look something like this (all values are hypothetical and only for understanding purposes):
|1. Testing schedule is tight. If the start of the testing is delayed due to design tasks, the test cannot be extended beyond the UAT scheduled start date.||High||High|
|2. Not enough resources, resources on boarding too late (process takes around 15 days.)||Medium||High|
|3. Defects are found at a late stage of the cycle or at a late cycle; defects discovered late are most likely be due to unclear specifications and are time consuming to resolve.||Medium||High|
|4. Scope not defined completely defined||Medium||Medium|
|5. Natural disasters||Low||Medium|
|6. Non-availability of Independent Test environment and accessibility||Medium||High|
|7. Delayed Testing Due To new Issues||Medium||High|
The final step in this Risk Based Testing (RBT) process is to find solutions to plan how to handle each one of these situations. These plans can differ from company to company, project to project and even person to person.
Risk Mitigation Techniques:
Here is an example of what the Risk’s table transforms into when this phase is complete:
Testing schedule is tight. If the start of the testing is delayed due to design tasks, the test cannot be extended beyond the UAT scheduled start date.
|High||High||• The testing team can control the preparation tasks (in advance) and the early communication with involved parties.
• Some buffer has been added to the schedule for contingencies, although not as much as best practices advise.
Not enough resources, resources on boarding too late (process takes around 15 days.
|Medium||High||Holidays and vacation have been estimated and built into the schedule; deviations from the estimation could derive in delays in the testing.|
Defects are found at a late stage of the cycle or at a late cycle; defects discovered late are most likely be due to unclear specifications and are time consuming to resolve.
|Medium||High||Defect management plan is in place to ensure prompt communication and fixing of issues.|
Scope completely defined
|Medium||Medium||Scope is well defined but the changes are in the functionality are not yet finalized or keep on changing.|
|Natural disasters||Low||Medium||Teams and responsibilities have been spread to two different geographic areas. In a catastrophic event in one of the areas, there will resources in the other areas needed to continue (although at a slower pace) the testing activities.|
|Non-availability of Independent Test environment and accessibility||Medium||High||Due to non availability of the environment, the schedule gets impacted and will lead to delayed start of Test execution.|
|Delayed Testing Due To new Issues||Medium||High||During testing, there is a good chance that some “new” defects may be identified and may become an issue that will take time to resolve.
There are defects that can be raised during testing because of unclear document specification. These defects can yield to an issue that will need time to be resolved.
If these issues become showstoppers, it will greatly impact on the overall project schedule.
If new defects are discovered, the defect management and issue management procedures are in place to immediately provide a resolution.
A few points to note:
This brings us to an end of Risk management in the Test planning phase. Even though the underlying steps and principle are similar, the risk management process is more concentrated towards the AUT when it happens in the test designing/execution phase.
In our next section we will cover – Risk management in test execution phase.
In our journey to understanding the risk management process, we talked about how it goes exclusively in the Test planning phase of risk-based testing. We also understood the generic process that involves – Risk identification, Risk Assessment and Risk Mitigation Plan.
How to Manage Risk at Test designing or Test execution phase:
There is one other form of Risk management (also sometimes referred to as, Risk-based testing) that occurs during the Test designing or Test execution phase depending on the situation. Now, what situation are we talking about? Let us try to understand that first.
We all know that our testing work is reactive. No requirements (or scope defined), we can’t perform a feasibility analysis and write test scenarios or plan for testing activities. Similarly, when the code is not ready we have nothing to test, no matter how much of prep-work we might have been ready with in terms of test cases, test data etc. Also, testing is the only step left before the product goes live.
Let us understand this better with an example:
If testing was to begin on a said date, Jan 1st and had to go on until Jan 14th – when the testing is done, the product’s Go-live date is usually fixed immediately. Let’s say- Jan 15th for simplicity’s sake. Now, in perfect world things would go exactly as planned. But we all know the reality.
In this case, let us assume that for some reason, the testing did not start until Jan 7th, which means we lost half of our testing time. But we do need 14 days to test the product thoroughly. We could move the go-live date farther by 7 days- however; this usually is not an option. Because the product is promised to hit the market on a certain date and any delays are not good for the business. So, usually, we testing teams have to absorb the delays, compensate somehow, work with the time available and make sure the product is tested well. Tough job, isn’t it?
This is where the risk management process is again employed.
What is the process?
Risk management takes place to determine what areas of the AUT (application under test) need maximum focus. These are typically the functional areas (modules or components) that are critical for the success of the final product and are most prone to failure.
Who performs it?
Since it is regarding the AUT, the knowledge about it is not only with the QA but with all the other teams – Dev, BA, Client, Project management teams etc. Therefore it is a collective effort, driven by the testing team.
Step #1: Risk Identification
Identify all the functional areas of the AUT. This will simply include making a list.
Step #2: Risk Assessment
All the risks are quantified and prioritized in this step. Quantifying is simply, assigning a number to each risk in the list that will give an indication of the priority with which it needs to be addressed.
Every risk’s probability (the chance of occurrence) and impact (amount of loss that it would cause when this risk materializes) are decided.
The typical method is to assign the ratings. For example, Probability can take values 1 to 5. 1 being the probability of occurrence being low (not likely to happen at all) and 5 being high (will most certainly happen).
Similarly, Impact can also be assigned a 1-5 rating. 1 being low impact (even if this risk does materialize, the loss is minimum) and 5 being high impact (huge losses when happens).
Make a table format and circulate to all the team reps- Dev, BA, Client, PM, QA and anyone else relevant.
Instruct each team to fill in the values based on their rating for probability and impact.
Since the probability and impact values are numeric it will make the calculating of the “Risk Factor” value easy.
Risk factor = Probability X Impact. The higher the risk factor, the serious the problem is.
Please note that at this point, this is simply the outcome of one team’s rating. For a project where 5 different teams are involved the QA team would end up with 5 different tables.
Take an average of the Risk factor values. For example, if there are 5 teams, for each module, add all the risk factor values and divide it by 5. This is the final values we are going to deal with. Say, these are the averaged risk factors:
The more the risk factor, the more that module has to be tested.
So, Module 5 and 2 are most crucial for the success of the business. Share the results with the all the teams.
Risk Mitigation Plan – Now, this is the step that changes from Project to Project. We have identified that modules 5 and 2 are the ones that need to be concentrated on most.
Examples of the plan could be:
Once a plan is made, all the teams reach an agreement and follow it to test the product keeping into account the risk-factor.
A few important points to note:
Risk Based Testing approach clearly indicates that the focus of the tester is not just to keep exploring the defects, irrespective of severity and priority. Now things have changed and Testers have to work smart and they need to understand the clear ‘Need of the Customer, and Wants of the User’.
They need to study the product thoroughly and understand which is the most frequently used feature in the production, which is the most critical path for revenue generation and how to protect and safeguard the customers from the production issues and business threats.
Hence, RBT approach clearly educates the3 testers that just testing everything or testing extensively does not mean that testing is complete or there are no defects in the product. Testing effectively in a stipulated time and ensuring that critical and major business impacts are nullified and that is quite important for the tester.
Thus Risk based testing is the most efficient tool for the QA team to guide the project stakeholders based on the project risks. RBT approach helps the QA team in the continuous identification of risk and its resolution that could endanger the achievement of the overall project goals and objectives and helps in achieving the ultimate aim of the QA Group.
P.S. The words QA and Testing have been interchangeably used throughout the document.
About the Author: This article is written by multiple STH team members – Gayathri Subrahmanyam and Swati S.
Gayathri is a Software Test SME with 2+ decades of experience in Software Testing and has extensively adopted ‘Risk Based Testing’ approach as a part of Test Industrialization in several engagements and has realized the benefit of Test resource optimization and quality testing.
Was Risk based testing a challenging one to you? Do you have any interesting facts to add on to our tutorial? Feel free to express your thoughts in the comments section below!!