Read this hands-on Devmate review including its pricing, features, and installation steps. Also, learn to create unit tests using Devmate Software:
Devmate is a tool or software utility to automatically generate unit tests for a given set of class files. Unit test coverage is a widely accepted metric and is deemed to be as high as possible for applications to ensure that the code is defect-free to some extent.
Having a good unit coverage, would ease out a developer as well as a QA team and ensure there are a basic set of tests already executed. Further with most of the applications being built in CI-CD pipelines, every commit or change to the application code would ensure running these tests and would in turn ensure that there are no regression defects being introduced as a result of the change.
devmate can be installed as a Visual Studio or IntelliJ plugin and can be used currently for C# and JAVA language.
Table of Contents:
Devmate Review
=>> Click here to Visit Devmate Website
Pricing and Download Info
Devmate has Single user and Corporate user licenses billed per month.
- Single user license: It costs around €9/month/per user. This is ideal for freelancers/individual developers.
- Corporate users: It costs around €49/month (for 2 to unlimited users) and has Dev support included.
There’s a free trial version as well which is valid for 10 test models per week.
=>> Click here to download the free version (after submitting some user and email details)
Note: It can also be downloaded from Microsoft Marketplace or Extension Manager in Visual Studio (IntelliJ coming in 7/2021).
Languages/Frameworks Supported
Currently, Devmate supports C# language and can be installed as a plugin in Microsoft Visual Studio. JAVA version as IntelliJ plugin is already in release and coming in 7/2021).
OS Requirements: Windows 10, with recommended 8 GB of RAM
With Devmate, you can generate unit test code for frameworks like
- nUnit
- xUnit
- msTest
- JUnit
All the above frameworks can be used to generate tests for .NET and JAVA-based applications.
Recommended Reading=>> Unit Testing Frameworks
Features
The features are as follows:
#1) Automatic Test case generation
- The generated test cases are 100% compliant with ISTQB and ISO standards.
- Generated test code is a data-driven, well-structured code.
- Complex data types are supported. In addition to primitive data types, complex data types are supported as well. For example: if the method under test expects a user-defined object, we can define the class data for such an object in the test model file.
- Persistent test data, the data used for the test cases is created as a data-driven parameterized unit test code and resides within the same repository (As opposed to an external data file like a database or excel/CSV file)
#2) Considerable reduction in Manual Testing effort
A lot of effort that goes into writing boilerplate code for testing is saved, as well as usage of Black box testing techniques ensures a lot of requirements specifications are validated through the automated test cases.
#3) Maintainable & Extensible
New tests can be added and the existing ones can be edited depending on the changes required in the current Application or Class file.
Testing Methodology
Unit testing is generally considered to be more of a white box testing where most of the developers would test the functionality of the code that they have written from a logic standpoint.
Devmate’s methodology makes use of Black box or Requirements based testing which treats the software or application under test as a black box (of which the system does not have knowledge about inner working). This makes validating the code more suited to the requirements as opposed to just checking for syntax errors. Hence, these tests would not only check logic but also in a way would ensure that the requirements are met.
Some black box testing techniques it could be using are:
Equivalence Class Partitioning: Here inputs are grouped into equivalence classes depending on which the test scenarios for a given function or application are constructed. An equivalence class is a range of values of an input variable for which it is expected that all values are treated in the same way/same behavior.
An example of an equivalence class can be a range of valid or invalid values.
Boundary Value Analysis: This technique extends equivalence partitioned classes and ensures that there is sufficient test coverage around the boundary values – as those are the areas where programmers tend to miss the validation and might cause serious issues with the Application.
Example: For a percentage value that is valid between 0 to 100 – boundary values would be
- Lower and upper boundaries – 0 and 100
- Values near the lower/upper boundaries – ex -1, 1 and 101, 99
Installation
The devmate comes as a vsix plugin and is downloaded as a .vsix file (Visual Studio Plugin)
Prerequisite: MS Visual Studio 2019 and above (Language supported – C#).
Remark: JAVA Language in IntelliJ is supported as of 7/2021.
Install Steps:
#1) Once you receive the download link for the devmate plugin, start the download and wait for the plugin download to complete.
#2) To start the installation of the plugin, double click the .vsix file downloaded from the above step.
#3) The installation will detect the available installations of Microsoft Visual Studio on your computer. (If not available the install will fail).
Please ensure that all open windows of Visual Studio are closed during the installation of the devmate plugin.
#4) The prompt is updated once the install process is complete.
#5) Open Visual Studio and check the devmate in the menu. (The extension should be available at – Extensions -> devmate)
You can also access other links – like Documentation and a small Intro video on using the devmate tool for automatic test case creation.
Creating Simple Tests Using Devmate
Creating the Test Application
#1) Create a project of type Class Library: named calculatorlib
#2) You will need an Application class that needs to be unit tested. Let’s create a calculator class that can perform different functions like Add, Subtract, etc. For this tutorial, we will create a simple function that returns the sum of 2 integers. Name the project as calculatorlib and create a class named Calculator.
Here’s the code for the Calculator class:
using System; namespace calculatorlib { public class Calculator { public int add(int input1, int input2) { return (input1 + input2); } } }#3) Create a test project: Add an NUnit-based test project to the solution. This will contain all the tests which we are going to generate.
#4) We are using an NUnit-based template so that all the requirements like NUnit test runner are taken care of. Name the project as autounittests
Generating Test data
#1) In order to generate automatic tests for a given function/class -> Navigate to the class – and Right-click the method for which we want to generate the tests.
In the menu as displayed below, select “Test with devmate”.
#2) You will be prompted to choose a location to place the devmate test model. Select an appropriate folder in the Application. This file will store all modeling data that the devmate needs to create the tests.
A test model file is a file with .tmdl extension and is the most important piece of using devmate software. This contains the data for the function that we want to test.
It contains info like:
- Input fields
- Output/expected values
- Exception details
#3) Notice the Devmate window opening within Visual Studio. It shows the UI for different test data and expected outputs.
Find out all the required Input/Output variables that would be needed in order to create different test scenarios.
#4) You can set the expected output/values against the different input combinations that are displayed.
Here, you can see that we have added different sets of values for input1 and input2 which would be used as inputs to our original “add” function in the “Calculator” class.
At this point, we are defining the possible values that our input data can contain.
The actual generated test can use the combination of these values.
#5) After adding test data, you can create “Generate Test Cases” to generate test scenarios against the data specified.
You will also need to update the “Expected Result” row with the result that we are expecting against the selected input values.
Please note that, in expected values, we can also assert exceptions if the application test code is expected to throw an exception against the given input values.
#6) Choose the framework for which the tests need to be created. The available options are – NUnit, xUnit, and MsTest
(Default selected framework is NUnit).
#7) Once the framework is selected and the button to generate test code is clicked. It will prompt for the location where we want to create the tests.
You can choose the test location in the NUnit-based test project that we created.
#8) Now, you will see a test class is created, and it has code/data for the test scenarios that we had specified.
In the above image, you can see the highlighted test model file which usually lives in your application project, and the actual tests file in the test project that we had created.
Executing The Test
- Execute the Tests by navigating to the “Test” menu in Visual Studio and click “Run Test”
- Once the command is selected, an NUnit test runner window will open and will start executing the generated test scenarios.
Please note that we can also run the tests by directly right-clicking on the source code file in Solution Explorer and selecting the “Run Tests” option.
Editing/Adding New Test Case
A lot of times, we want to add a new test or edit an existing test, depending on changes to the original class.
If we want to change the expected output for one of the tests, we can
- Click on the Model file, which opens the devmate visual editor with the test cases.
- Update the test data/test name or add the expected output depending on the requirements.
- Re-generate the test cases – You will be prompted to Overwrite or Merge the tests. Overwrite is generally a good option if you don’t have any manual changes that were done to the file.
- Once the test code is re-generated, we can try running the tests again. Since we updated the Expected output for => -3 + 5 to 10 (instead of the actual returned by function i.e. 2) – our test is expected to fail.
In the above image, you can see – 4 tests are passing and the one for which we updated the expected output is now marked as failed.
Similar to editing the expected output for an existing test, we can also add a new test case or remove any existing one. The auto-generated test names are like – p1, p2, etc – These can also be updated as desired.
So, that’s how you could update things like – test data or test name or add a new test to an existing test model that was created through the devmate test model.
Note: The test class is generated using an Auto Code generator. You can modify the contents but should not modify the header comments and a code-generated hash at the end of the file. Doing that can corrupt the file and might make the modifications to the devmate tests difficult.
Pros And Cons
Pros
- Easy to use – Since it’s integrated as an extension with Visual Studio, you don’t need to move to a different tool/application for test case generation. It’s closely integrated with Visual Studio (which is the de-facto standard for developing .NET based applications)
- Reduces Dev effort on testing dramatically up to ~50% – Reduction in boilerplate code and allows the developer to focus on providing the set of input and output data with all the redundant code/parameterized classes etc is taken care of by the devmate code generator.
- Support for Multiple test frameworks – like NUnit, xUnit, and MsTest.
- Artificial Intelligence and Machine Learning Support – It learns from the existing tests to suggest and create new, more robust tests.
Cons
- Is a licensed tool – it adds an additional cost component.
- Limited languages are supported. Currently, just C# (based on the Website – Java support would be launched soon)
Conclusion
In this tutorial, we learned about the Devmate automated testing solution which comes as an integrated plugin for the Visual Studio Tool. It makes the repeated Unit testing and even complex scenarios like branches/loops to have unit tested easily, by creating automated code for Unit tests.
It helps developers by reducing the boilerplate/repeated effort of defining test inputs/names etc but is intelligent enough to identify input/output parameters and what tests should be required for a particular class or method under test.
In addition to this, it has AI and ML integration, which enables the devmate software to learn from its recommendations and keep improving in terms of better test scenario suggestions.