This tutorial explains in detail about Automated Fault-Tolerance Testing Using Katalon Studio. Let’s get started.
Have you read the book Fail Fast, Fail Often: How Losing Can Help You Win by Ryan Babineaux and John Krumboltz? It teaches us to have a unique look at failures, know our level of tolerance to faults made, and then pick ourselves up with grace.
However, in practice, putting out a requirement for flawless releases or a defect-free testing process is unrealistic. Instead, minor bugs that bring little harm to the application functionality can be bypassed.
This way, teams can redirect their efforts and attention to more severe issues, such as disruptions and malfunctions in the system as a whole.
If a component, system, or software can work as it was designed to in the presence of the defects, then those faults can be tolerated.
Table of Contents:
Automated Fault-Tolerance Testing
It goes without saying: The sooner defects and failures are spotted in the SDLC, the less costly they are to fix.
Following the concept of shift-left in software testing, gaining early and frequent feedback can save both money and work. As bugs inch their way toward production, the cost of fixing increases since they are now more connected to other codes.
Errors are now complex to reproduce, take up development time, and ultimately fail teams to deliver releases or applications on time.
What software development teams often do are:
- Make use of Continuous Integration (CI) and static analysis to keep code clean.
- Involve QA engineers in the initial stages to understand requirements and architecture.
- Perform Root Cause Analysis to identify if defects were caused by a single condition or a causal chain.
- Learn from these failures and predict software behaviors to decrease fault tolerance and increase system reliability.
Determine a Failure Threshold
Defining a level of failure threshold depends on the size of your test suite, the number of test cases, and the failure patterns your team has observed from past runs. Just look at how teams do Sprint Planning and story point estimation.
The velocity at which one team can go from sprint to sprint cannot apply to another, since it is a local measurement. Team A might have members who hold more experience in handling complex features, and adversely, team B might come with an uneven range of technical expertise.
As a result, team B’s number of story points will be greater than team A’s, as they will need more time to handle them. Additionally, reordering a list of issues in the backlog for current and future sprints can also be necessary.
For this reason, giving an accurate estimation of the failure threshold can be done in several ways, such as:
- Add up the total test cases of the tests you will be running.
- Finding the average number of failed tests in the last 90 days.
Having these two measurements determined, you will know the point of failure to jump in and investigate the root causes without having to wait until every single test has finished.
For example, if your test volume stands at 1,000 test cases, you could put out a threshold of 10%. This means that if the number of failed test cases reaches the 100th, then that limit of tolerance would cancel the execution of the following tests.
For larger projects, the volume of tests can reach the hundredth and thousandth and test listeners can only do so much. Working with frameworks like Selenium, your tests will run through one by one without the ability to alert that many have failed.
In such cases, early feedback on your flawed piece of code prevents you from having to sit and wait for hours for the entire test suite to complete, only to know that there were problems right from the start.
Note: We’ll use Katalon Studio’s Terminate Execution Conditionally to see how this works.
Stop Test Suite Execution After ‘X’ Failed Tests In Katalon Studio
Terminate Execution Conditionally
1) Set a failure threshold for your Test Suites
First, launch Katalon Studio and have your project, test suites, and Katalon Runtime Engine ready.
*Note: Katalon Runtime Engine (KRE) is needed to run your test in CLI mode.
Click on the Build Command button on the left of the Playback button.
Select your Organization and tick Terminate the execution once the total number of test failures reaches this threshold box.
Enter your desired number and click Generate Command.
This means that once Katalon Studio has detected the ‘x’ number of failed test cases you have defined, your test suites will stop running immediately.
2) Run Test Suites from command-line
Click Copy to Clipboard.
Then, paste the command with the -maxFailedTests=T parameter into your Terminal and press Enter.
3) Execution terminated and Pass/Fail tests
Once the failure limit has been reached, you will see this message, as shown below.
You can have an overview of the Passed and Failed tests for every run.
View Pass/Fail Trends in Katalon TestOps
Now that you have decided on your failure threshold, connect to TestOps, Studio’s built-in test reporting and orchestration platform, to look at your pass and fail trends.
1) Set up an Organization and Project
First, log in to analytics.katalon.com with your Katalon account.
2) Turn on TestOps integration in Katalon Studio
In Katalon Studio, go to Project > Settings (applies for both Mac and Windows).
- Click on Katalon TestOps > check the Override authentication box > select your Organization (if you cannot find your Organization, click Fetch Organization)
- Input your Katalon email and password.
- Tick Enable Katalon TestOps Integration > choose your Team > Project > Apply and close.
3) Push and view Katalon Studio Test Suite Results on TestOps
Note: TestOps will gather all your test runs using Test Suite. Make sure to have your Test Cases organized in a specific Test Suite for executions to be recorded fully.
Locate TestOps on the left menu > Test Runs > choose a Test Run.
Once you navigate to TestOps, you will be able to view the pass/fail patterns.
Other test reports like Flaky and Stale reports are also available if you want to explore a bit more.
Start your first project at Katalon Studio.
Need more information about the Fail-fast principle? See the full article for Terminate Execution Conditionally in Katalon Studio.
Suggested read =>> Browser automation testing for Start-ups
What do you think of this article? Please provide your feedback in the comments section below. We would love to hear from you.