JUnit Test Suite & Filtering Test Cases: JUnit 4 Vs JUnit 5

This Tutorial will discuss What is a JUnit Test Suite, How to Create a Test Suite and How to Filter Test Cases in JUnit 4 vs JUnit 5:

We learned about how to skip certain test cases during execution in our previous tutorial. We also learned about different annotations used in JUnit 4 and JUnit 5 to do so.

In this tutorial, we will learn:

• What is a test suite?
• How do we create a test suite by encasing multiple test cases into a group and executing the suite using JUnitCore class?

=> Visit Here For The Exclusive JUnit Training Tutorial Series. 

JUnit Test Suite

JUnit 4: @RunWith, @SuiteClasses Annotations

The tutorial Multiple ways to Execute JUnit Tests illustrated how to create a test suite in JUnit 4.

The annotations @RunWith and @SuiteClasses helped us to create a test suite by grouping multiple JUnit test classes. Subsequently, a runner file with class JUnitCore.runclasses() invoked the execution of the test suite created.

Please refer to the mentioned section for all the minute details on the workflow along with the actual code for JUnit 4.

JUnit 5: @RunWith, @SelectClasses, @SelectPackages Annotations

The creation of a test suite in JUnit 5 is quite similar to what we have in JUnit 4. So, where is the difference then?

#1) In JUnit 4, we have Suite.class which is passed as a parameter to @RunWith annotation to support the creation of a test suite whereas JUnit 5 uses the same @RunWith annotation but with the input parameter as JUnitPlatform.class instead of Suite.class.

So, the line of code in JUnit 5 looks like @RunWith(JUnitPlatform.class). This is the annotation that comes packaged with your subproject JUnit Platform.

#2) In JUnit 4, we use @SuiteClasses to group multiple JUnit classes separated by a comma while in JUnit 5 we have:

• The annotation @SelectClasses that is equivalent to @SuiteClasses in JUnit 4 to group multiple JUnit classes.
• @SelectPackages annotation is used to group multiple tests from the package(s). You need to input a string array value which represents the package you wish to include.

Thus, in other words,

1. If in case you want to group test cases from a single package, JUnit 5 enables you to do that.
2. Or in case, you wish to group test cases from multiple packages, JUnit 5 supports you do that as well. A note to remember here is, that the tests under all the sub-packages of the mentioned package also get included in the test suite by default.

JUnit 5: Different Scenarios/Examples

Creating A Test Suite grouping Multiple Test Classes

The snippet of the code is shown below:

@RunWith(JUnitPlatform.class)
@SelectClasses({JUnit5TestCase1.class, JUnit5TestCase2.class })
public class JUnitTestSuite { }

Creating A Test Suite for Single Package

The snippet of the code is shown below:

@RunWith(JUnitPlatform.class)
@SelectPackages({“demo.tests“})
public class JUnit5TestSuite { }

Note:

1. Say demo.tests package has a subpackage demo.tests.subtests.
2. The code @SelectPackages({“demo.tests“}) will include all the tests under the subpackage, too, into the test suite; by default.
3. If you had stated @SelectPackages({“demo.tests.subtests“}), the test cases under the subpackage demo.tests.subtests will only be included in the test suite whereas the tests from its parent package i.e. demo.tests would not be included.

Creating A Test Suite For Multiple Packages

The code snippet for creating a test suite for multiple packages separated with a comma in JUnit 5 – will look as shown below:

@RunWith(JUnitPlatform.class)
@SelectPackages({“demo.tests”, “demo1.tests”, “demo2.tests”})
public class JUnit5TestSuite { }

Filtering Test Suite – JUnit 4 vs JUnit 5

Sometimes, we need to filter out test cases and execute a particular set of tests per our needs. For instance, there could be a set of tests identified for regression testing, a separate set for unit testing, and a separate set of test cases for smoke testing.

We need to exclude or include test cases from certain packages or classes or categories. Filtering out or tagging the tests from a single package is the only option with JUnit4.

Compared to JUnit 4, JUnit 5 comes with a good set of features on supporting this need of filtering your test cases from the single package or multiple packages and sub-packages per your changing requirements.

JUnit 4 – @Category, @IncludeCategory, @ExcludeCategory

The test cases in JUnit 4 can be categorized into different categories. These categories can be excluded or included while running your test suite. The annotations @Category, @IncludeCategory, and @ExcludeCategory are supported from JUnit 4.12 and above.

High-level steps to create the filters based on the category or categories are as follows:

#1) Create a marker Interface that plays a role in the category.

#2) Annotate the test methods of the classes to be included in the SuiteClasses with @Category and the category name.

#3) In the test suite file, use annotation @IncludeCategory with the category name to include the tests belonging to a specific category.

#4) In the test suite file, use annotation @ExcludeCategory with the category name for excluding them.

#5) The @Category annotation can be used at the test level or class level as well. If the annotation is applied at the test level, then that specific test gets tagged with the given category whereas if the annotation is at the class level, all the tests within the class are tagged to the given category.

Let’s look at some more details to understand the practical implementation of categorizing the tests and filtering them out for execution:

Step 1:

We will start with the creation of a marker interface that will play the role of a category for test methods. Here, we create a category named UnitTest. The code is very simple. Please refer to the code below.

Code for UnitTest.java

package demo.tests;
public interface UnitTest {}

Step 2:

We shall mark the category at the test method level in the class JUnitTestCase1.java. To add the test case junitMethod1() into the category UnitTest, we need to annotate the test method with @Category(UnitTest.class).

This adds the test method to the UnitTest category. The other test methods (if any) are not tagged to the category unless the methods are annotated with a category.

To have @Category annotation working in our code, we need to import the package org.junit.experimental.categories.Category

The code snippet from JUnitTestCase1.java:

 @Category(UnitTest.class)
 @Test
public void junitMethod1(){
int Value2=9000;
Assert.assertEquals(Value1, Value2);
}

Alternatively, a test method can belong to multiple categories too E.g.: @Category(UnitTest.class, SmokeTest.class)

Step 3:

I would now, mark the category at the class level in the class JUnitTestCase2.java. The same statement that was added at the test method level in the earlier step will be added in the current class file as well.

Note that here, we will add the statement on the class level. Doing this will make all the test methods in the file belong to the UnitTest Category.

The code snippet from JUnitTestCase2.java:

 @Category(UnitTest.class)
 @Test
public class JUnitTestCase2 {
public String stringValue="JUnit";

Step 4:

Now that our required test cases have been categorized into the UnitTest category, we will now see how to add them into the test suite by filtering them out based on the category. We shall make certain code changes in the JUnitTestSuite.class to demonstrate this.

• Categories.class will be passed as a parameter to the @RunWith annotation.
• @Suite.SuiteClasses will take the test classes string array.
• The annotation @Categories.IncludeCategory will need UnitTest.class as the parameter.
• This statement will help us to filter the entire suite and run only those test cases from the suite that belongs to the category.
• Categories.class needs package org.junit.experimental.categories.Categories to be imported.

The code snippet for JunitTestSuite.java

@RunWith(Categories.class)
@Categories.IncludeCategory(UnitTest.class)
@Suite.SuiteClasses({JUnitTestCase1.class, JUnitTestCase2.class})
public class JUnitTestSuite {

The annotation @Categories.IncludeCategory can also be written as @IncludeCategory. You may also create multiple interfaces (categories) and annotate the class/test methods with multiple categories separated by a comma. The below example will filter out the tests belonging to category – Category1, and Category2.

Example: @IncludeCategory({Category1.class, Category2.class})

Similar rules go with @Categories.ExcludeCategory / @ExcludeCategory to exclude the test methods under the category or categories during the test run.

JUnit 5 – @IncludeTags, @ExcludeTags, @IncludePackages, @ExcludePackages, @IncludeClassNamePatterns, @ExcludeClassNamePatterns

JUnit 5 comes packaged with multiple approaches for organizing and filtering out the test cases.

JUnit 5 – @IncludeTags, @ExcludeTags

#1) Like JUnit 4 has @IncludeCategory and @ExcludeCategory annotations to support filtering test cases for execution.

#2) JUnit 5 has @IncludeTags and @ExcludeTags annotations to achieve the same purpose.

#3) JUnit 4 refers to the test cases to be organized into a specific category while JUnit 5 refers to tagging the test cases with a specific tag to enable filtering the test cases for execution.

High-level steps to create the filters based on the Tags are as follows:

1. Annotate the test methods of the package(s) to be included in the @SelectPackages with @Tag and the user-defined tag name. One class can have different tags for different test methods.
2. You may also annotate @Tag at the class level so that all the tests in the class are tagged.
3. In the test suite file, use annotation @IncludeTags with the tag name to include the tests belonging to a specific tag.
4. In the test suite file, use annotation @ExcludeTags with the tag name for excluding them from the test suite.

Let’s now have a detailed illustration on how to practically implement filtering in JUnit 5.

Step 1: We are tagging a test method in JUnit5TestCase1.java to the Tag name “Regression”

The code snippet from JUnit5TestCase1.java:

@Tag(“Regression”)
 @Test
public void junitMethod1(){

Step 2: We are tagging a test method in JUnit5TestCase2.java to the Tag Name “SmokeTest”.

The code snippet from JUnit5TestCase2.java:

@Tag(“SmokeTest”)
@Test
public void junitMethod2(){

Step 3: Now that the test methods have been tagged, we will now update JUnit5TestSuite.java to add appropriate filters by tags for the tests. The below code includes all tests that are tagged as ‘Regression’ and exclude all those tagged as ‘SmokeTest’.

The code snippet from JUnit5TestSuite.java:

@RunWith(JUnitPlatform.class)
@SelectPackages({“demo.tests“})
@IncludeTags(“Regression”)
@ExcludeTags(“SmokeTest”)
public class JUnit5TestSuite { }

JUnit 5 – @IncludePackages, @ExcludePackages

We already know that when we pass a package name in the @SelectPackages annotation, the tests in the sub-packages of the package also get added to the test suite.

There could be certain sub-packages that we want to include in our test suite while a few other sub-packages that we do not want or might not be relevant to be included in our suite.

This is facilitated through the annotations @IncludePackages and @ExcludePackages in JUnit 5.

Let’s assume that we have a package ‘demo.tests’ which has three sub-packages i.e. subpackage1, subpackage2, and subpackage 3 with its test classes in each of the below packages.

Let’s see a code snippet of JUnit5TestSuite.java to visualize how to include and exclude a package.

Scenario #1: Apply filter to include test cases from subpackage1 only.

The below code includes all the tests from all the JUnit classes in the package demo.tests.subpackage1, however, excludes all the tests directly under the package demo.test and those under the package subpackage2 and subpackage3.

The code snippet from JUnit5TestSuite.java:

@RunWith(JUnitPlatform.class)
@SelectPackages({“demo.tests“})
@IncludePackages(“demo.tests.subpackage1”)
public class JUnit5TestSuite { }

Scenario #2: Apply filter to exclude test cases only from package subpackage3.

The below code excludes all the tests from the JUnit classes in the package – demo.tests.subpackage3 however the suite includes all the tests directly under the package demo.test and those under the package subpackage1 and subpackage2.

The code snippet from JUnit5TestSuite.java:

@RunWith(JUnitPlatform.class)
@SelectPackages({“demo.tests“})
@ExcludePackages(“demo.tests.subpackage3”)
public class JUnit5TestSuite { }

JUnit 5 – @IncludeClassNamePatterns, @ExcludeClassNamePatterns

When you want to include or exclude certain classes matching a specific regular expression from the package, the annotations @IncludeClassNamePatterns and @ExcludeClassnamePatterns can be used in the test suite class file.

Let us now see the illustration through the code update in JUnit5TestSuite.java

Scenario #1:

The below code includes classes that end with ‘Ctests’ from the package demo.tests

@RunWith(JUnitPlatform.class)
@SelectPackages({“demo.tests“})
@IncludeClassNamePatterns({"^.*CTests?$"})

Scenario #2:

We shall now exclude those classes that initiate with ‘STest’ from the package demo.tests

@RunWith(JUnitPlatform.class)
@SelectPackages({“demo.tests“})
@ExcludeClassNamePatterns({"^STest.*$"})

Scenario #3:

Multiple regular expressions also can be passed as filter criteria. The below code states two different regular expressions separated with the ‘OR” operator. The slit bar | is the OR operator here.

The code filters out classes initiating with Test (Test.*) or ending with Tests (*. Tests?) and includes those classes in the test suite as annotation @IncludeClassNamePatterns is used.

Note: If the fully qualified class name matches against at least one pattern, then the class will be fetched per the @IncludeClassNamePatterns or @ExcludeClassNamePatterns annotation.

@RunWith(JUnitPlatform.class)

@SelectPackages({“demo.tests“})

@IncludeClassNamePatterns({“^(Test.*|. *Tests?) $”})

FAQ About JUnit Filtering Test Cases

Here is a question that might have been lingering on your mind for quite some time now.

Q #1) What is the difference between disabling/ignoring a test and filtering out a test? Can’t either of the features help to serve the same purpose of skipping test cases during execution?

Answer: The question indeed is a genuine one and is worth answering.

1. With the filtering out test feature, you may exclude as well as include tests depending on the tagged category. Whereas, with disabling, you may decide only on excluding and not including tests for execution.
2. Filtering out the test cases is a kind of conditionally skipping test case whereas, in the case of Ignored tests, those are skipped with no conditions.
3. Another significant difference between the two is that when you run the test cases with tests annotated with @Ignore or @Disabled – the ignored tests show in the test results under the SKIPPED count
4. When you run the filtered-out tests, they do not show up in the test result at all.

Conclusion

In this tutorial, we learned how to filter out tests by adding a category/tag to the tests.

We learned in JUnit 4, that we have @Category, @IncludeCategory, and @ExcludeCategory for filtering the test cases while JUnit 5 has @IncludeTags and @ExcludeTags to do the same.

Besides, JUnit 5 has additional filtering options using annotations @IncludePackages, @ExcludePackages, and also annotations to include or exclude classes using class name patterns. The more we are exploring; we realize that there is much more yet to explore.

=> Watch Out The Simple JUnit Training Series Here.