The Ultimate Guide to Risk Based Testing: Risk Management in Software Testing

The Ultimate Guide To Risk Based Testing, Risk Management, and Its Approach with Examples:

What is Risk Based Testing?

Risk based testing is to carry out testing or to design and execute the scenarios, such that the top business risks which will have a negative impact on the business as identified by the customer are unearthed in their product or feature early in the life cycle and are mitigated by implementing mitigation measures. 

Risk based Testing

The negative impact can include cost impact, dissatisfied customer, bad user experience, and even to the extent of losing the customers.

In other words, RBT approach is to ensure that, testing is done in a such a way that even if a user finds a bug in the production, that does not stop him/her from using the software or does not impact the business in a serious way.

RBT is testing carried out based on the product risks. RBT is to find out well in advance, as what is the risk of the failure of a particular feature or functionality in the production and its impact on the business in terms of cost and other damages by using a prioritization technique for the test cases.

Hence, Risk based testing uses the principle of prioritizing the tests of the features, modules, and functionalities of a product or software. The prioritization is based on the risk of likelihood of failure of that feature or functionality in production and its impact on the customers.

Risk Based Testing and Its Relevance to Agile & DevOps

Three hundred hours spent on developing a software can be made useless in just 30 seconds with a single defect identified in production.

This, in turn, can ruin the purpose of the whole product with no other option but just to withdraw it from the market. And that is the significance and need for ‘Quality testing’.

With the rapid growth in technology, software is hosted on the cloud supporting multiple OS, multiple platforms, complex IT infrastructures, etc., the end users are becoming more and more fussy about the features, options and they never compromise for customer satisfaction.

Nowadays, ‘Quality’ is becoming a crucial factor in software delivery, where continuous improvements are happening to improve the quality in order to keep the customers happy.

We often notice that it is a common problem for almost all the Testers to be under tremendous pressure that their testing window is squeezed and typically the build is handed over to them for testing at the last minute. There is no enough time and resources for them to run all the tests that they have designed and the automation coverage is not always 100% and it has its own challenges.

Factors of Risk based testing

RBT Lack of time

The delivery timeline cannot be missed and at the same time quality cannot be compromised as well. Whatever the plan B, to add additional test resources by pulling out from the other teams, is not working out, Plan C, stop doing all the other activities and divert the effort to this alone, is not really helping. However much addition of test resources, at the end, is not working out.

RBT Less time

There is no other option but just to run limited and important tests within the available time and resources.

So, how do we decide which test is important at this stage? Whatever a Tester considers important may not be really important for the customers. From whose perspective is the importance of the feature or a functionality decided? Who will decide which are the important tests? And a lot of other questions keep arising.

In order to answer all these questions and to handle the above situation effectively, a testing approach called ‘Risk Based Testing’, shortly called ‘RBT’, came into existence, where the team has clearly planned and identified the test scenarios based on the ‘Project Risk’ criteria.

Though the QA team has a clear picture of the important tests, RBT is a proven method of identifying the crucial and important tests from the customer and business perspective through a ‘Risk Analysis’ procedure.

So, now as against the traditional way of simply identifying the defects in the software, the QA approach and goal has changed with the time due to the change in the technology, increase in the competition in the market to release a quality software, introduction of ‘Automate everything’, and in totality introduction of Agile and DevOps practice of delivering the software over a period of few hours.

Hence, the current trend of  ‘Testing Principle’ is not just merely ‘identifying the defects’ but to also,

#1) Focus on the area of the product where there is a high impact on business due to failure or high likelihood of failure in the production.

#2) Focusing on identifying the defects early and allowing a team to fix it as early as possible and hence allowing the software/product or feature to ‘Fail Fast’.

#3) The most important aspect of Service of the QA team now is to focus on the customer in bringing value to the customer by increasing the focus on ‘end to end customer experience’.

Risk Based Testing Approach

It is always like preparing for an examination, that one cannot say testing is enough and there are no more defects in the software, even if they design and execute an ample number of tests.

There is a point at which software stability is not going to be doubly assured by increasing the number of test cases alone. At this point of time, it is not just to focus upon the number of tests but on what actually the customer is expecting from the release.

Hence, it is essential to strike a balance in optimizing the testing in order to get the maximum benefit with the reasonable effort of testing. And this is more important when the testing timelines are very tight and enough resources are not available to carry out sufficient testing.

Thus, in this case, RBT approach plays a key role in optimizing the QA effort and maximizing the testing benefit with the minimum business risk.

So, if we focus on the above aspect, then the work of a QA will be much relieved. They do not have to burn their weekends in the office, continuously testing the software and getting worried about every S4 (Severity 4) and P4 (Priority 4) defects that come out of the testing.

Well, 4 is considered as lowest priority and severity of the defects in testing. They can better invest their time in other useful aspects of the project.

To summarize the key drivers of the ‘Risk-based testing’ approach:

#1) To enable testing ‘what customers want’ from a business perspective.

#2) To meet the time schedule with expected quality.

#3) To optimize the QA effort.

When do we use RBT Approach?

This is used under the below scenarios:

  • RBT approach can be used whenever there is a limitation or constraint on Time, cost and resources of a project and whenever there is a need to optimize the resources.
  • RBT approach is used when the program is more complex and adapts new technology and hence involves a lot of challenges.
  • When the program is an R&D project and it is of first kind type and there is a number of unknowns and risks in the project.

RBT Approach Example

Several risk based analysis approaches are used in the IT industry to overcome the risks faced by the production and its impact.

Given below is one such approach:

This approach of RBT includes identifying the ‘Vital Functionalities or Key Features’ of the product and assessing the risks to which each of these functionalities get exposed to in the production and implementing appropriate mitigation measures in place to lower or mitigate the risk.

Hence the RBT approach includes testing the functionalities which have the probability of failure and highest impact on business. The types of failures could be operational or business, technical, external etc.

Risk based analysis approach

Ways to carry out Risk Analysis

There is no standard process or template defined as such to carry out the risk analysis in Software testing for each and every feature of a product. Various organizations use their own approach of risk analysis methods.

Risk analysis can be carried out on various project items to identify the risks and to implement RBT approach for analysis. Those items include,

  • Features
  • Functionalities
  • User Stories
  • Requirements
  • Use Cases
  • Test Cases

In this case, let us focus only on Test cases to understand the Risk Based Testing approach implementation.

Risk Analysis Procedure

Risk analysis includes the involvement of relevant stakeholders of the program from both the ‘Technical Team and Business Team’, which includes, Owner of the Product, Product Managers, Business Analysts, Architects, Testers, and Customer Representatives.

Brainstorming session involving these stakeholders would be organized to carry out the discussion to identify the importance of each feature of a product and prioritize them based on the risk of failure and its impact to the end users in production.

Various ‘Project Documents’ such as Requirements Document, Technical Specification Documents, Architecture and Design Documents, Business Process Document, Use Case Document etc., will become the input for the brainstorming session.

Stakeholder’s knowledge about the product and the existing product in the market will also be an input factor for the discussion.

Few other sources of inputs can also include,

  1. To gather inputs on most used functionality.
  2. Inputs from consulting a domain expert.
  3. Data from the previous version of the product or similar product in the market.

During the brainstorming session, the risks pertaining to each of these features are identified. The types of risks could be an operation, technical or business related. The tests and scenarios related to them are weighted and risk values are assessed based on the likelihood of risk occurrence and impact of the risk.

The ‘Likelihood of risk occurrence’ can be due to:

  1. Poor understanding of the feature by the product development team.
  2. Improper architecture and design.
  3. Insufficient time to design.
  4. Incompetency of the team.
  5. Inadequate resources – people, tools, and technology.

The ‘Impact of the risk’ is the effect of failure to the users and business if it occurs. The impact could be,

  1. Cost impact, resulting in a loss.
  2. Business impact resulting in losing business or losing market share, Law proceedings, Loss of license.
  3. Quality impact, resulting in substandard or incompetent product release.
  4. Bad user experience, resulting in dissatisfied and loss of a customer.

The focus area of assessing the risk of a feature or product can be,

  • Area of Business criticality of the functionality.
  • Most used features and important functionality.
  • Defect prone areas
  • Functionality bearing the safety and security impact.
  • Area of Complex Design and Architecture.
  • Changes made from previous versions.

Risk Analysis Methodology

Let us understand the ‘Risk based testing methodology’ in detail now.

The risk based testing approach uses RISK as the criteria at all test phases of the testing cycle, i.e., test planning, test design, test implementation, test execution and test reporting. Ideally, one can design a numerous number of possible test scenario combinations.

Hence, the RBT approach includes a ranking of the tests based on the risks severity to find out the most defective or risky area of failure, which causes high impact to the business.

The main objective of Risk analysis is to distinguish between the ‘High value’ items like product features, functionalities, requirements, user stories, and test cases, and ‘Low Value’ ones and hence later to more focus on ‘High value’ Test Cases, by less focusing on ‘Low value’ Test Cases. This is the initial step of Risk analysis before starting the risk based testing.

The main task of Categorization or grouping of Test Cases into High Value & Low Value and assigning the priority value to each of these test cases includes the following steps:

Step #1: Using a 3X3 grid

Risk analysis is performed using a 3X3 grid, where each functionality, non-functionality and its related Test cases are assessed by a team of stakeholders for its ‘Likelihood of failure’ and ‘Impact of failure’.

Risk Analysis

Likelihood of failure of each functionality in the production is generally accessed by a group of ‘Technical Experts’ and are categorized as ‘Likely to fail, quite likely and unlikely’ along the vertical axis of the grid.

Likelihood of failure

Similarly, the ‘Impact of failure’ of these features and functionalities in production is experienced by the end customer, if not tested is assessed by a group of Business Specialists” and are categorized under ‘Minor, Visible and Interruption’ categories along the horizontal axis of the grid.

Impact of failure

Step #2: Likelihood and Impact of failure

All the Test cases are positioned in the quadrants of the 3 X 3 grid based on the identified values of a likelihood of failure and impact of failure which are shown as dots in the picture below.

Likelihood and impact of failure

Obviously high likelihood of failure and high impact of failure (interruption) are grouped in the top right corner of the grid, which is of high importance and hence it is identified that ‘High Value’ tests and ‘Low Value’ tests are grouped in the bottom left corner which is of least or no importance to the customer, where minor focus can be given to these features or test cases.

Step #3: Testing Priority Grid

Based on the above positioning of the test cases in the grid, the tests are prioritized and labeled with priorities 1,2,3,4 and 5 and are marked against each of them. The most important tests are positioned in a 1st grid are assigned with priority 1 and similarly less important ones are ranked as 2, 3, 4 and 5.

Finally, all the test cases are sorted based on their priority numbers and are picked up for execution in the order of priority. The high priority ones are picked up for execution first and low priority ones are either executed later or de-scoped.

Testing Priority grid

Step #4: Details of Testing

The next step is to decide on the level of details of testing for the defined scope of testing. The depth of scope of the testing can be decided based on the above ranking as per the below grid.

High priority tests with ranking 1 are ‘More Thoroughly’ tested and accordingly, experts are deployed to test this high criticality features and its related Test Cases. Similarly test cases with priority 2, 3, and 4. A decision to de-scope priority 5 features and tests based on the available time and resources can be taken.

Level of Detail of Testing

Hence, Level of Detail of Testing approach of prioritizing the features and its test cases not only helps the Testers to identify the ‘High Value Tests’ but also guides them to decide on their ‘detail level of testing’ based on these priority rankings and helps them to carry out better testing and reduces testing cost by optimization process.

How is RBT Relevant to Agile and DevOps?

Now, after understanding the Risk Based Testing approach of carrying out the testing based on the prioritization of tests depending upon the ‘Risk of Failure’ of a particular feature and its ‘Impact to Customer’ in live, obviously one would raise the question of the relevance of Risk Based Testing approach in Agile and DevOps Practices.

‘Automate Everything’, ‘Test Everything’, ‘Test Continuously’, ‘Test Repeatedly’ are the key concepts of these practices.

Each time, when there is a change in the code or there is a release, all the designed tests are run through the automated Continous integration(CI) / Continuous delivery(CD) pipeline quickly and repeatedly, irrespective of their prioritization.

Then what is the relation between RBT and DevOps? Where would RBT fit in and become relevant in Agile and DevOps???

Relevance of RBT for Agile and DevOps

#1) Yes, as I said earlier, it is not that every Industry and every product has 100% automation coverage for their test executions. So, if at all the team has to make the choice of prioritization for test execution from the choice of manual test cases and would like to spare the energy and effort of the test resources towards other activities, then RBT is the best choice.

Risk based approach is also a better bet for running automated tests with high priority ones and testing at the earliest.

#2) QA team can adopt RBT approach more effectively during the Requirement Analysis in analyzing the requirements and providing an advance report of the probable risks of the products and the features so that appropriate actions could be taken proactively by the program team to mitigate it.

#3) RBT approach can be effectively used in designing the Test Cases and scenarios based on High risk so that more focus can be paid towards high-risk features and functionalities.


#4) Identifying the ‘High Risk’ areas enables the QA team to focus their Testing effort on those areas to test ‘More thoroughly’ using ‘High Skilled Testers’.

#5) ‘Fail Fast’, as we all know is the concept of ‘Agile’ and ‘DevOps’, for which RBT approach helps to identify the ‘high risk’ areas in the software, identify the defects early and allow them to fail fast and fail first and let the team fix it.

#6) The ultimate goal of Agile/DevOps is ‘Customer focus’ and hence RBT approach enables the QA to focus on Customer experience than just finding Defects.

Risk based testing in Devops and Agile

Benefits of Risk Based Testing approach

We already understood the purpose and use of RBT approach of analyzing the requirements, designing and executing the testing scenarios. There are several benefits of RBT.

We can consolidate and list the benefits of Risk Based Testing as:

  • Helps in more efficient and optimized use of Test Resources.
  • Helps in easing the QA work, Testing, Test Design & Development and Test Preparation activities by Prioritising.
  • Helps in better managing the QA resources by allocating the key resources towards high focus areas.
  • Helps in effective utilization of resources and diverts their time and energy on better things in the project.
  • Helps the QA team in planning the testing efforts based on the risk assessment and identification of volatile and high-risk areas.
  • Helps the team to optimize the tests to be conducted depending on the importance and hence de-scope low-value tests in agreement with the stakeholders.
  • Overall it helps in Cost reduction through optimized and reduced testing activities.
  • RBT approach enables the QA team to test the high-risk areas first and allows the product to ‘Fail fast’ and fix fast.
  • RBT approach helps in bringing a clarity in the ‘Test Coverage’ and the ‘Test Scope’ to the entire group of Stakeholders.
  • The team can increase their focus on High-risk areas and keep less focus on the low-risk area.
  • RBT allows the Team to decide well in advance on the implementation of a most effective way of mitigation for the product risks.
  • RBT Helps in avoiding the effect of inappropriate implementation of mitigations.
  • Risk based testing allows the team to take appropriate action either to mitigate or to plan for contingency or to position any workaround to overcome the failure or reduce its impact if the risk occurs in Live.
  • RBT helps in minimizing the residual risk in the release.
  • Helps in achieving the ‘Improved Quality’ through less costly defects in the production.
  • Ultimately helps in ‘Improved customer experience’ and ‘Satisfied Customer’.

Next, we will learn how to manage risks at Test Planning and Test execution phases of Software Testing Life Cycle.

***************************************************************

Risk Management During Test Planning

 

How to Manage Risks During Test Planning Phase:

Life is full of risks, and so is a software project. Anything can go wrong anytime. We are always on our toes to make things right – but what about making sure that nothing goes wrong and that when it does we know exactly what to do? Enter Risk management – this is a portion of a software testing project that prepares us to prevent, understand, find and get over risks.

A risk is simply a problem that is likely to occur and when it does occur, will cause a loss.

The loss could be anything- money, time, effort or a compromise in quality. The loss is never good. No matter how much of a spin we give it,it’s not positive- and never will be. Therefore Risk management is an integral part of software projects to make sure we handle risks and prevent/reduce losses.

how to manage risks during test planning

Risk-Based Testing: Since we are a QA community let us stay specific to risks and the process related to it in our QA world exclusively. Risks are assessed and managed roughly in 2 phases of our Software Test life cycle. STLC can be categorized into 3 phases -Test planning, Test designing and Test execution.

The risk management process occurs twice, during:

  1. Test planning
  2. Test case design(end) or sometimes in the test execution phase

It is mandatory in case 1 but with case 2 it is more of a ‘need-basis’ situation.

risk based testing process

Two part article series:

Even though the underlying process is the same, the types of risks addressed in both these areas are completely different. In order to do justice to them individually, we are going to handle them differently as a two-part series.

This section is going to be about “Risk Management during test planning”.

Risk Management Process

The generic process for Risk management involves 3 important stages:

  1. Risk identification
  2. Risk impact analysis
  3. Risk mitigation

Testing Risks and Mitigation Examples:

#1) Risk identification

 

As it is said, the first step to solving a problem is identifying it. This stage involves making a list of everything that might potentially come up and disrupt the normal flow of events.

The main outcome of this step is a list of risks.

This risk-based testing step is commonly led by the QA lead/Manager/representative. However, the lead alone will not be able to come up the entire list- the entire QA team’s input makes a huge impact. We can say this is a collective activity led by the QA lead.

Also, the risks that are identified during the Test planning phase are more ‘managerial’ in orientation- meaning, we are going to look at anything that might impact the QA project’s schedule, effort, budget, infrastructure changes, etc. The focus here is not the AUT, but the way the QA phase will go on.

Risks During Test Planning: Risk Based Testing Examples

The following is a sample list of risks that might be listed during test planning phase. Please note, that the AUT and its functionality is not the focus here.

#1. The testing schedule is tight. If the start of the testing is delayed due to design tasks, the test cannot be extended beyond the UAT scheduled start date.

#2. Not enough resources, resources onboarding too late (process takes around 15 days.)

#3. Defects are found at a late stage of the cycle or at a late cycle; defects discovered late are most likely be due to unclear specifications and are time-consuming to resolve.

#4. Scope not defined completely defined

#5. Natural disasters

#6. Non-availability of Independent Test environment and accessibility

#7. Delayed Testing Due To new Issues

At this point, you can choose to be as thorough as you would like, depending on the amount of time available.

Once, all the risks are listed, we progress to Risk assessment/Risk impact analysis.

#2) Risk assessment/Risk impact analysis

 

Is your Risk Analysis something like this? :)

risk management cartoon

Risk Analysis in software testing: All the risks are quantified and prioritized in this step. Every risk’s probability (the chance of occurrence) and impact (amount of loss that it would cause when this risk materializes) are determined systematically.

High – medium – low, values are assigned to both the probability and impact of each risk. The risks with “high” probability and “High” impact are taken care of first and then the order follows.

Risk impact analysis table:

Following these steps, the risk impact analysis table for the above risks listed would look something like this (all values are hypothetical and only for understanding purposes):

RiskProbabilityImpact
1. Testing schedule is tight. If the start of the testing is delayed due to design tasks, the test cannot be extended beyond the UAT scheduled start date.HighHigh
2. Not enough resources, resources on boarding too late (process takes around 15 days.)MediumHigh
3. Defects are found at a late stage of the cycle or at a late cycle; defects discovered late are most likely be due to unclear specifications and are time consuming to resolve.MediumHigh
4. Scope not defined completely definedMediumMedium
5. Natural disastersLowMedium
6. Non-availability of Independent Test environment and accessibilityMediumHigh
7. Delayed Testing Due To new IssuesMediumHigh

#3) Risk Mitigation

 

The final step in this Risk Based Testing (RBT) process is to find solutions to plan how to handle each one of these situations. These plans can differ from company to company, project to project and even person to person.

Risk Mitigation Techniques:

Here is an example of what the Risk’s table transforms into when this phase is complete:

RiskProb.ImpactMitigation Plan
SCHEDULE
Testing schedule is tight. If the start of the testing is delayed due to design tasks, the test cannot be extended beyond the UAT scheduled start date.
HighHigh• The testing team can control the preparation tasks (in advance) and the early communication with involved parties.
• Some buffer has been added to the schedule for contingencies, although not as much as best practices advise.
RESOURCES
Not enough resources, resources on boarding too late (process takes around 15 days.
MediumHighHolidays and vacation have been estimated and built into the schedule; deviations from the estimation could derive in delays in the testing.
DEFECTS
Defects are found at a late stage of the cycle or at a late cycle; defects discovered late are most likely be due to unclear specifications and are time consuming to resolve.
MediumHighDefect management plan is in place to ensure prompt communication and fixing of issues.
SCOPE
Scope completely defined
MediumMediumScope is well defined but the changes are in the functionality are not yet finalized or keep on changing.
Natural disastersLowMediumTeams and responsibilities have been spread to two different geographic areas. In a catastrophic event in one of the areas, there will resources in the other areas needed to continue (although at a slower pace) the testing activities.
Non-availability of Independent Test environment and accessibilityMediumHighDue to non availability of the environment, the schedule gets impacted and will lead to delayed start of Test execution.
Delayed Testing Due To new IssuesMediumHighDuring testing, there is a good chance that some “new” defects may be identified and may become an issue that will take time to resolve.
There are defects that can be raised during testing because of unclear document specification. These defects can yield to an issue that will need time to be resolved.
If these issues become showstoppers, it will greatly impact on the overall project schedule.
If new defects are discovered, the defect management and issue management procedures are in place to immediately provide a resolution.

A few points to note:

  1. The sooner risk management starts in a QA project planning phase, the better.
  2. Of all 3 steps, risk identification is the most important. If anything is not listed and considered for further steps, the risk goes unhandled.
  3. Try to find an ideal time frame for this activity. Remember, too much planning leaves too little time for doing.
  4. Also, after the risk management process, if a new situation comes up, the risk management plan can be altered or updated to reflect the most current condition.
  5. Historical data can be very useful for the success of this process.

Conclusion

This brings us to an end of Risk management in the Test planning phase. Even though the underlying steps and principle are similar, the risk management process is more concentrated towards the AUT when it happens in the test designing/execution phase.

In our next section we will cover – Risk management in test execution phase.

*******************************************************

Risk Management at Test Execution Phase (with Example)

 

In our journey to understanding the risk management process, we talked about how it goes exclusively in the Test planning phase of risk-based testing. We also understood the generic process that involves – Risk identification, Risk Assessment and Risk Mitigation Plan.

How to Manage Risk at Test designing or Test execution phase:

There is one other form of Risk management (also sometimes referred to as, Risk-based testing) that occurs during the Test designing or Test execution phase depending on the situation. Now, what situation are we talking about? Let us try to understand that first.

We all know that our testing work is reactive. No requirements (or scope defined), we can’t perform a feasibility analysis and write test scenarios or plan for testing activities. Similarly, when the code is not ready we have nothing to test, no matter how much of prep-work we might have been ready with in terms of test cases, test data etc. Also, testing is the only step left before the product goes live.

Risk Management – with a focus on the AUT

Risk based testing part 2

Let us understand this better with an example:

If testing was to begin on a said date, Jan 1st and had to go on until Jan 14th – when the testing is done, the product’s Go-live date is usually fixed immediately. Let’s say- Jan 15th for simplicity’s sake. Now, in perfect world things would go exactly as planned. But we all know the reality.

In this case, let us assume that for some reason, the testing did not start until Jan 7th, which means we lost half of our testing time. But we do need 14 days to test the product thoroughly. We could move the go-live date farther by 7 days- however; this usually is not an option. Because the product is promised to hit the market on a certain date and any delays are not good for the business. So, usually, we testing teams have to absorb the delays, compensate somehow, work with the time available and make sure the product is tested well. Tough job, isn’t it?

This is where the risk management process is again employed.

  • Now, if delays are anticipated ahead of time before even the testing begins- the process takes place in the Test designing phase.
  • If delays happen during a test execution phase that started normally- the process is followed during the test execution phase.
  • The steps and the method are the same no matter what stage it happens.

What is the process?

Risk management takes place to determine what areas of the AUT (application under test) need maximum focus. These are typically the functional areas (modules or components) that are critical for the success of the final product and are most prone to failure.

Read also => Failure mode and effects analysis (FMEA) is a risk management technique

Who performs it?

Since it is regarding the AUT, the knowledge about it is not only with the QA but with all the other teams – Dev, BA, Client, Project management teams etc. Therefore it is a collective effort, driven by the testing team.

How does Risk Bases Testing take place?

Step #1: Risk Identification

Identify all the functional areas of the AUT. This will simply include making a list.

Step #2: Risk Assessment

All the risks are quantified and prioritized in this step. Quantifying is simply, assigning a number to each risk in the list that will give an indication of the priority with which it needs to be addressed.

Every risk’s probability (the chance of occurrence) and impact (amount of loss that it would cause when this risk materializes) are decided.

The typical method is to assign the ratings. For example, Probability can take values 1 to 5. 1 being the probability of occurrence being low (not likely to happen at all) and 5 being high (will most certainly happen).

Similarly, Impact can also be assigned a 1-5 rating. 1 being low impact (even if this risk does materialize, the loss is minimum) and 5 being high impact (huge losses when happens).

Step #3:

Make a table format and circulate to all the team reps- Dev, BA, Client, PM, QA and anyone else relevant.

Risk based testing 1

Step #4:

Instruct each team to fill in the values based on their rating for probability and impact.

Since the probability and impact values are numeric it will make the calculating of the “Risk Factor” value easy.

Risk factor = Probability X Impact. The higher the risk factor, the serious the problem is.

An example:

Risk based testing 2

Please note that at this point, this is simply the outcome of one team’s rating. For a project where 5 different teams are involved the QA team would end up with 5 different tables.

Step #5:

Take an average of the Risk factor values. For example, if there are 5 teams, for each module, add all the risk factor values and divide it by 5. This is the final values we are going to deal with. Say, these are the averaged risk factors:

Risk based testing 3

The more the risk factor, the more that module has to be tested.

So, Module 5 and 2 are most crucial for the success of the business. Share the results with the all the teams.

Step #6:

Risk Mitigation Plan – Now, this is the step that changes from Project to Project. We have identified that modules 5 and 2 are the ones that need to be concentrated on most.

Examples of the plan could be:

  1. Modules 5 and 2 will be tested thoroughly by making sure all the test cases related to them are tested. The other modules will be tested on an exploratory basis.
  2. Modules 5 and 2 are going to be tested first and then depending on the time available the others are going to be taken care of.

Once a plan is made, all the teams reach an agreement and follow it to test the product keeping into account the risk-factor.

That’s it!

A few important points to note:

  • Since this is a collective activity that takes everyone’s opinion into consideration; the chances of it being accurate and effective are higher.
  • This is not a formal method and does not have to be a part of every QA project.
  • Sometimes, even if the team chooses not to draw tables and assign values- a simple brainstorming session with everyone present can give the QA team good direction on how to proceed.
  • The development team’s input is very important because they are the ones who create the product, so they will know what might work and what might need additional checking. Be sure to be on the lookout for that.
  • Even though there are multiple steps in the process, it does not take a considerable amount of time to perform them. Especially, if all the teams can sit together and work simultaneously.
  • Remember this process and its outcome is only the alternative. Getting as much time as planned for testing is the best way to perform the QA activity.

Conclusion

Risk Based Testing approach clearly indicates that the focus of the tester is not just to keep exploring the defects, irrespective of severity and priority. Now things have changed and Testers have to work smart and they need to understand the clear ‘Need of the Customer, and Wants of the User’.

They need to study the product thoroughly and understand which is the most frequently used feature in the production, which is the most critical path for revenue generation and how to protect and safeguard the customers from the production issues and business threats.

Hence, RBT approach clearly educates the3 testers that just testing everything or testing extensively does not mean that testing is complete or there are no defects in the product. Testing effectively in a stipulated time and ensuring that critical and major business impacts are nullified and that is quite important for the tester.

Thus Risk based testing is the most efficient tool for the QA team to guide the project stakeholders based on the project risks. RBT approach helps the QA team in the continuous identification of risk and its resolution that could endanger the achievement of the overall project goals and objectives and helps in achieving the ultimate aim of the QA Group.

P.S. The words QA and Testing have been interchangeably used throughout the document.

About the Author: This article is written by multiple STH team members – Gayathri Subrahmanyam and Swati S. 

Gayathri is a Software Test SME with 2+ decades of experience in Software Testing and has extensively adopted ‘Risk Based Testing’ approach as a part of Test Industrialization in several engagements and has realized the benefit of Test resource optimization and quality testing. 

Was Risk based testing a challenging one to you? Do you have any interesting facts to add on to our tutorial? Feel free to express your thoughts in the comments section below!!





30 comments ↓

#1 smita

which is more important risk management during planning or execution?

#2 Suresh

risk management is a continuous process throughout the SDLC and STLC. But it would have more positive impact when done at proper stages.

#3 Zakir

Good post Swati.
Keep up the good work !!!

@smitha

Need to wait and stay tuned to the next article on Risk management in test execution phase which can answer your question :)

#4 kote

Pardon me for my poor english. Where it says “Holidays and vacation have been estimated” in the example of mitigation, what exactly does it mean? Does it mean considering working on holidays and vacation if the risk occured?

#5 Swati Seela

@smita: Both are important and really worth the time. RM at Test planning is almost compulsory.

#6 Swati Seela

@Suresh: A great observation. Thank you for sharing it with us

#7 Swati Seela

@kote: nope, it means planning has been thorough to include holidays and vacation times – this means schedule variance(if any)will occur only due to unforeseen issues, but not due to slip-ups during planning. Hope that makes sense.

#8 Amruta Wable

Thank you for this post,its really nice and understand to read risk testing,can you please send me some points about SDLC

#9 kote

@Swati: sorry for not writing back sooner. I’ve been away in a while with limited Internet access. Thank you for the elaboration. I now have a clear idea!

#10 Rajahmix

This was the wonderful post, it taught me a lot about risk management especially during test planning process.

#11 Rajesh

Hi,

The article is good. I have few points to add and how do we mitigate them?. Also, I suggest to identify the maximum risks during test planning phase and document them in Test Plan before requirements review is completed.
1. Schedule risk – requirements got delayed and it has subsequent effect on next tasks like development design/coding, testing.
2. Test Environment Risk – Coding is completed and build is ready to be deployed in test areas but test areas are not yet setup.
3. Resources Risk – The resources planned for testing have resigned from organization during the peak testing phase and they carryout lot of functional knowledge.
4. Communication Risk – There is an communication misunderstanding/problem among requirement engineers, development and testing teams.
5. Product Risk – If two simultaneous releases are planned to merge at the end of testing phase. Due to the issues identified in release that affects the other – the product is under RISK if gets released. The management has to take decision on it.

Do let me know your comments on this.

#12 Swati Seela

@Rajesh: It is a great question. Risk analysis and planning is a preventive activity but it does not mean it will remove the risk altogether. So, all the situations you mentioned are possible in which we have an issue.

Most issues in IT projects are collective decisions. When these issues do come up- it means its time to sit together and come up with a alternate plan or workaround.

Risk management prepares us to an extent, but the rest of it is still up to the forces of nature :)

#13 Srinath

Its great explanation for @ Swati Seela

#14 Ramesh Bestha

Its great explanation of Risk management for Testing

#15 Sarita Bennett

Outstanding posting; thanks Swati! Thanks Rajesh for your comments and points. I too thought of identifying the maximum risk during the initial test plan and modify throughout the SDLC…

#16 sarvesh

Hi,
very good post on Risk management. Thanks.
Could I request you to add more in Mitigation plan ? I can see risk/problem is again explained in mitigation plan. Mitigation is to resolve the problem if occurred. i.e. Scope not defined, environment not available, changing requirements.

Thanks,

#17 Vinod

Thanks a lot Swati, this article was really helpful.

#18 paedru

Everything is wonderfully explained.

Except for – How to look out for risks?
The answer is – Where to look for for risks?(The areas to focus upon in a software implementation)

The below points out those areas to focus upon to look for risks instead of randomly thinking out areas of risks which could appear in a software project.

1. Functionality
2.Localization
3.Usability
4.Relaibility
5.Performace
6.Suportability and so on.

Once the areas are listed for a project,within the area determine in whihc particular functionlaity things could go worng(by asking about complexity of the function to the tech team) and impact on business to the Clients.Then multiply those factors by asiiging wieghtage and mitagate those risks.

#19 ben

Great Article..Thx

#20 ben

Here goes some more Risks

1.Poor requirement
2.Non-availability of SRS/BRS
3.Inexperienced or unskilled resources
4.Poor client communication
5.Delay from the development phase.

#21 Rupesh

Hi Swati, It was a good article. But still, I cant get to 1 question:
How to calculate cost for a risk? Be it any risk, resource on leave etc.

#22 Princess

This is a really good article for Test Managers, Test Leads and Testers to read. Thanks a lot for all the contributes. It’s a rich article for me , as am currently writing a Test Strategy and about writing about the risk.

#23 Sar Sophara

I think it’s very useful and helpful to me.
And I would like to recommend to those who like reading the articles, especially relevant to Risk Management.

Thanks

#24 Sengu

Great Article….

#25 Ashish

Hi
I have question,

If someone not having the latest version of build then what would be the mitigation Plan for this

#26 Narendra

Hi Swathi,

Very well explained.

Thanks,
Naren

#27 yeshwanth

is there a program of how to calculate risk

#28 mR Big dIcK

shut up jacob

#29 js

Great Article..

#30 Sanjiv Patel

@Rajesh Please explain more on test areas are not yet setup.
I came a cross ALM (QC) not setup correctly, Project and column not found in report.

Leave a Comment