JUnit Test Execution Order: Order Of Tests JUnit 4 Vs JUnit 5

This Tutorial Explains how to Set the Execution Order for JUnit Test Cases. You will learn about Annotations & Classes to set the JUnit Test Execution Order in JUnit 4 vs JUnit 5:

We learned how to create a test suite, how to add a category or tag to the test cases, and how to filter out tests (exclude or include the test cases) based on the category or tag in our previous tutorial.

Besides, we learned that in JUnit 4, we have @Category, @IncludeCategory, and @ExcludeCategory for filtering the test cases while JUnit 5 has @IncludeTags and @ExcludeTags to do the same.

JUnit 5 has additional filtering options using annotations @IncludePackages, @ExcludePackages, and also annotations to include or exclude classes using class name patterns.

=> Check Here To See A-Z Of JUnit Training Tutorials.

JUnit Test Execution Order

In this tutorial, we will learn how to set an order for each test case so that they run in the set order. We will learn how to order tests in JUnit 4 as well as in JUnit 5.

The test methods don’t follow a specific order by default. The test cases need not necessarily execute in the order in which they have been written.

There are different ways or modes to set the order of execution for the test cases. We will also deep dive into how JUnit 4 varies in ordering the test cases when compared to JUnit 5.

JUnit 4: @FixedMethodOrder, Class MethodSorters

Starting JUnit 4.11, we have the annotation @FixMethodOrder and MethodSorters.class supporting the facility of setting an order for a test’s execution.

The package org.junit.runners.* needs to be imported to include the class MethodSorters. This class decides how the test cases need to be ordered. MethodSorters have three enum values.

Given below are the enum values of the class along with the purpose each of the value serves:

The annotation @FixedMethodOrder takes in the input parameter of MethodSorters with its enum value. The package needed for the annotation is org.junit.FixedMethodOrder.

Let’s see how it is implemented through the code.

Code Implementation for MethodSorters.DEFAULT

Let’s create a JUnit class file ‘Junit4TestOrder.java’ where we will use MethodSorters.DEFAULT

The code for Junit4TestOrder.java

@FixMethodOrder(MethodSorters.DEFAULT)
public class JUnit4TestOrder { 
@Test
    	public void Testcase_3() {
        		System.out.println("Testcase_3 executes");
    	}
	 @Test
	 public void Testcase_1() {
	        	System.out.println("Testcase_1 executes");
	    }
	 @Test
	  public void Testcase_2() {
	        	System.out.println("Testcase_2 executes ");	    
} }

We run the class thrice and see the same order of tests being run as below however, the test cases may or may not be executed in the sorted order.

The console window showed the results as below – the order of test execution is TestCase_3, TestCase_1, and TestCase_2 and the order doesn’t change with the number of runs.

Code Implementation for MethodSorters. JVM

We will now update ‘Junit4TestOrder.java’ to have MethodSorters.JVM

The code snippet for Junit4TestOrder.java

@FixMethodOrder(MethodSorters.JVM)
public class JUnit4TestOrder {

We run the class twice and the order of tests’ execution changes with each run.

The console window for the first run is as shown below:

The console window for the second run is as shown below:

Keenly observe the change in the order in which the tests are executed in both the runs. The order of tests in the two runs have differed.

Code Implementation for MethodSorters.NAME_ASCENDING

We will now update ‘Junit4TestOrder.java’ to have MethodSorters.NAME_ASCENDING

The code snippet for Junit4TestOrder.java

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class JUnit4TestOrder {

We run the class twice, the order of tests executed remains the same and is executed per the ascending order of the method name.

The console window shows the result of the tests executed in the order TestCase_1, TestCase_2, and TestCase_3.

JUnit 5: @TestMethodOrder, @Order, Interface MethodOrderer

To control the order of execution of tests, the below entities help do the same:

1. The annotation @TestMethodOrder
2. The annotation @Order
3. The classes that belong to MethodOrderer interfaces

The built-in MethodOrderer classes and their details are as below:

Let’s now look at the demonstration of each of these ordering strategies:

Code Implementation For Alphanumeric.class

Let’s create a JUnit 5 class file named JUnit5TestOrder.java same as JUnit4TestOrder.java and let’s use the annotation with Alphanumeric.class to order the tests alphanumerically.

The code for Junit5TestOrder.java

@TestMethodOrder(Alphanumeric.class)
public class JUnit5TestOrder { 
@Test
    	public void Testcase_3() {
        		System.out.println("Testcase_3 executes");
    	}
	 @Test
	 public void Testcase_1() {
	        	System.out.println("Testcase_1 executes");
	    }
	 @Test
	  public void Testcase_2() {
	        	System.out.println("Testcase_2 executes ");	    
} }

We ran the class thrice and still see the same order of tests being run in the alphanumerically sorted order of the test method name.

Post execution of the class file, the order of the test execution:

1. Testcase_1,
2. Testcase_2 and
3. Testcase_3

Note: The Alphanumeric sort strategy is case sensitive so in case we had another test case with the name testcase_1.

The order of execution would be:

1. Testcase_1,
2. Testcase_2,
3. Testcase_3,
4. testcase_1.

Hence, the upper case takes priority over the lower-case method names.

Code Implementation For Random.class

We will now update the JUnit 5 class JUnit5TestOrder.java to use the annotation with Random.class

The code snippet for Junit5TestOrder.java

@TestMethodOrder (Random.class)
public class JUnit5TestOrder {

We ran the class twice and we saw that every time, we ran the class, the order of the test execution was randomly ordered.

Post execution of the class file for the first time, the order of test execution was:

1. Testcase_2,
2. Testcase_1
3. Testcase_3

The order of execution when ran for the second time showed:

1. Testcase_2,
2. Testcase_3
3. Testcase_1.

Code Implementation for OrderAnnotation.class

We will now update the JUnit 5 class JUnit5TestOrder.java to use the annotation with OrderAnnotation.class. The annotation @Order will also play an important role in the priority setting of the test methods here.

The code snippet for Junit5TestOrder.java

@TestMethodOrder(OrderAnnotation.class)
public class JUnit5TestOrder { 

@Test
              @Order(1)
    	public void Testcase_3() {
        		System.out.println("Testcase_3 executes");
    	}

	 @Test
               @Order(2)	 
               public void Testcase_1() {
	        	System.out.println("Testcase_1 executes");
	    }
	 @Test
@Order(3)
	  public void Testcase_2() {
	        	System.out.println("Testcase_2 executes ");	    
} 
}

So, in this strategy of setting up the order for test execution, @Order annotation enforces the test methods to run per the order value being set for it.

The lower the value of the @Order for a test method, the higher is its priority during execution.

Post execution, the order of tests ran as follows:

1. Testcase_3,
2. Testcase_1
3. Testcase_2 because the order set for the test cases is 1,2 and 3, respectively.

This is why it doesn’t matter if the test cases are written in order. Besides, it doesn’t matter if the method names are or are not in alphanumeric order.

JUnit 5: Creating Custom Order

Besides, the built-in Order classes, JUnit 5 also supports custom orders by implementing the interface MethodOrderer. Starting from JUnit 5 version 5.4, the custom sort is supported.

Let’s quickly see how to create and implement a custom order by method length.

Step 1: Created a custom order class implementing MethodOrderer interface and named the class as TestCaseLengthOrder

The code for TestCaseLengthOrder.java

public class TestCaseLengthOrder implements MethodOrderer {
 @Override
public void orderMethods(MethodOrdererContext context) {
MethodDescriptor md1;
MethodDescriptor md2;
context.getMethodDescriptors().sort((md1, md2)->
     md1.getMethod().getName().length().compareTo(md2.getMethod().getName().length()));
    }
}

TestCaseLengthOrder code explanation:

1. A custom order class is created by implementing the MethodOrderer interface.
2. void orderMethods(MethodOrderContext context){ } method that is the in-built method implemented from the Interface MethodOrderer. Here is where you define the implementation of the test ordering logic.
3. MethodDescriptor is an interface that encloses details about a method:
   • The method MethodDescriptor.getMethod() gets the method name for this descriptor.
   • The method name is converted to String with the method getName() i.e. MethodDescriptor.getMethod().getName() and
   • The method length() fetches the length of the method (just like string.length() fetches length of a string value).
   • All the method names are compared to each other using the compareTo() method.
4. The method getMethodDescriptors() gets the list of all the method descriptors in a class.
5. The sort() method sorts the MethodDescriptor objects.

Now, that we have understood each API of MethodOrderer clearly, we hope that it is easy to interpret the above code.

Step 2: Use the custom order class like you use any built-in order in the test class.

As an input to the annotation @TestMethodOrder.

The code for JUnit_CustomOrder.java

@TestMethodOrder(TestCaseLengthOrder.class)
class JUnit_CustomOrder{
@Test
public void subt(){
}
@Test
public void add(){
}
@Test
public void multiply(){
}
@Test
 public void divide(){ }

Step 3:

Post execution of JUnit_CustomOrder.class, the order of test execution is as follows based on the ascending order of the length of test cases’ name:

1. add(),
2. subt(),
3. divide()
4. multiply()

Conclusion

To conclude this tutorial on JUnit Test Execution Order.

1. We learned how to set the order of test cases using specific annotations as well as specific classes.
2. We also learned various ways to order tests for JUnit 4 and JUnit 5, based on which the ordering strategies changed.
3. Besides, we learned how in JUnit 5, we could also create a customized sorting class and use it for ordering the test cases during their execution.

=> Take A Look At The JUnit Beginners Guide Here.