This tutorial explains Automated Fault-Tolerance Testing for your test suites using maxFailedTests in Katalon Studio and TestOps. Learn to fail fast and fail often now.
What is Fault Tolerance?
Have you read the book Fail Fast, Fail Often: How Losing Can Help You Win by Ryan Babineaux and John Krumboltz?
Well, in short, it teaches us to have a unique look at failures, knows our level of tolerance to faults made, and gracefully pick ourselves up. In practice, putting out a requirement for flawless releases or a defect-free testing process is not realistic. Instead, minor bugs that bring little harm to the application functionality can be bypassed.
This way, teams can redirect their efforts and attention to more severe issues, such as disruptions and malfunctions on the system as a whole. If a component, system, or software can work as it was designed to in the presence of the defects, then those faults can be ‘tolerated’.
What You Will Learn:
Automated Fault-Tolerance Testing
It goes without saying: The sooner defects and failures are spotted in the SDLC, the less costly they are to fix.
Following the concept of shift-left in software testing, gaining earlier and more frequent feedback can save both money and work.
As bugs inch their way towards production, the cost of fixing mounts since they are now more connected to other codes. Errors are now complex to reproduce, take up development time and ultimately failing teams to deliver releases or applications on time.
What software development teams often do are:
- Make use of Continuous Integration (CI) and static analysis to keep codes clean.
- Involve QA engineers in the initial stages to understand requirements, architecture.
- Perform Root Cause Analysis to identity if defects were caused by a single condition or a causal chain.
- Learn from those failures and predict software behaviors to decrease fault tolerance and increase system reliability.
Determine A Failure Threshold
Defining a level of failure threshold depends on the size of your test suites, the number of test cases, and the failure patterns your team has observed from past runs. Just look at how teams do Sprint Planning and story point estimation.
The velocity at which one team can go from sprint to sprint cannot apply to another, since it is just a local measurement, not global. Team A might have members who hold more experience in handling complex features, and adversely, team B might come with an uneven range of technical expertise.
As a result, team B’s number of story points will be greater than team A’s, as they will need more time to handle. Additionally, reordering the list of issues in the backlog for current and future sprints can also be necessary.
For this reason, giving an accurate estimation of the failure threshold can be done in several ways, such as:
- Adding up the total test cases of the tests you will be running.
- Finding the average number of failed tests in the last 90 days.
Having these two measurements determined, you will know the point of failure to jump in and investigate the root causes without having to wait until every single test has finished.
For example, if your test volume stands at 1,000 test cases, you could put out a threshold of 10%. This means that if the number of failed test cases reaches the 100th, then that limit of tolerance would cancel the execution of the following tests.
For larger projects, the volume of tests can reach the hundredth and thousandth–and test listeners can only do so much. Working with frameworks like Selenium, your tests will run through one by one without the ability to alert that many have failed.
In such cases, earlier feedback on your flawed piece of code prevents you from having to sit and wait hours for the entire test suite to complete, only to know that there were problems right from the start.
Note: We’ll use Katalon Studio’s Terminate Execution Conditionally to see how this works.
Stop Test Suite Execution After ‘X’ Failed Tests In Katalon Studio
Terminate Execution Conditionally
Step 1: Set the failure threshold for your Test Suites
First, launch Katalon Studio and have your project, test suites, and Katalon Runtime Engine ready.
*Note: Katalon Runtime Engine (KRE) is needed to run your test in CLI mode.
Click on the Build Command button on the left of the Playback button.
Select your Organization and tick the Terminate the execution once the total number of test failures reaches this threshold box.
Enter your desired number and click Generate Command.
This means that once Katalon Studio has detected the ‘x’ number of failed test cases you have defined, your test suites will stop running immediately.
Step 2: Run Test Suites from command-line
Click Copy to Clipboard.
Then, paste the command with the -maxFailedTests=T parameter into your Terminal and press Enter.
Step 3: Execution terminated and Pass/Fail tests
After the failure limit has been reached, you will see this message, as shown below.
You can have an overview of the Passed and Failed tests of every run.
View Pass/Fail Trends In Katalon TestOps
Now that you have decided on your failure threshold, connect to TestOps, Studio’s built-in test reporting, and orchestration platform, to look at your pass and fail trends.
Step 1: Set up an Organization and Project
First, log in to analytics.katalon.com with your Katalon account.
Step 2: Turn on TestOps integration in Katalon Studio
In Katalon Studio, go to Project > Settings (applies for both Mac and Windows).
- Click on Katalon TestOps > check the Override authentication box > select your Organization (if you cannot find your Organization, click Fetch Organization)
- Input your Katalon email and password.
- Tick Enable Katalon TestOps Integration > choose your Team > Project > Apply and close.
Step 3: Push and view Katalon Studio Test Suite Results on TestOps
Note: TestOps will gather all your test runs by Test Suite. Make sure to have your Test Cases organized in a specific Test Suite for executions to be recorded fully.
Locate TestOps on the left menu > Test Runs > choose a Test Run.
Once you are navigated to TestOps, you can view the pass/fail patterns.
Other test reports like Flaky and Stale reports are also available if you want to explore a bit more.
Start your first project in Katalon Studio.
Need more information about the Fail-fast principle? See the full article for Terminate Execution Conditionally in Katalon Studio.