Specflow Report Generator Tutorial: Ways to Generate Test Reports and Execute Selective Tests in Specflow:
In this Specflow Training Tutorials for All, we explored all about Generating Living Documentation with Pickles in our previous tutorial.
In this tutorial, we will discuss the various techniques to create Reports for the tests written in Specflow framework.
The type of report generated also largely depends upon the type of test runner that is being used. Both free and paid variants of runners are available with each having their own pros and cons.
We will be discussing more on creating reports using a combination of NUnit as test runner and Specflow reporting templates to generate HTML reports out of the NUnit generated test output files.
Table of Contents:
Specflow Reporting
Watch the VIDEO: Watch the Video tutorial on Specflow Reporting
Generating Test Reports – Prerequisites
Before starting with generating a test report, let’s ensure that all the prerequisites are available or installed.
We will use the same test example of YouTube search which uses Specflow template with NUnit test runner. In order to start generating reports, you will need the NUnit console to execute the Specflow tests.
Install NUnit console by downloading the executable package file for windows here. Download NUnit-Console -3.9.0 MSI package and complete the step by step installation of NUnit Console by running the downloaded package file.
Once installed successfully, you should be able to see the Nunit folder in the ProgramFiles folder.
Refer to the screenshot below:
Specflow report generation also works with other frameworks like MSUnit in the same way as NUnit.
Step by Step Process
#1) Execute the tests using NUnit Console to create an XML report.
- Compile the SpecflowProject in Visual Studio and ensure that there is no compile errors project dll file created. The created DLL file should be available in the Debug/Release folder of the Visual Studio Project.
- Now Open the Command Prompt and Navigate to the base location of the NUnit Console.
- Run the following command in order to execute the Specflow tests.
Let’s see the syntax first:
nunit3-console.exe "result={PathToReportLocation};format=nunit2" {SpecflowProjectDllPath}
Once the above command is executed, you will see all your scenarios in the feature files to start getting executed. After the test completion, ensure that the XML result file from the NUnit execution gets generated at the location specified in the result parameter in the above command.
Also please note here that the format of the generated report that we have is selected as “NUnit2” as the current version of Specflow does not support NUnit 3. Once this support is available, the report format option can be removed.
#2) Now use the specflow.exe application to create the HTML report.
The specflow.exe application runs over the XML report that’s generated by NUnit and generates good looking HTML reports. specflow.exe is an executable application present in the npm package folder of the project where specflow was installed.
See the screenshot below for locating the Specflow Executable.
The report generated by this application is of HTML format and contains a summary of executed Scenarios grouped by features.
Execute the following steps to generate an HTML report out of the XML report that got generated through NUnit console execution.
- Open the command prompt and navigate to the Specflow executable.
- Now run the command with the below syntax to generate the HTML report.
specflow.exe nunitexecutionreport --ProjectFile {PathToProjectSln.csproj} --xmlTestResult {PathToNUnitReportXml} --OutputFile {DesiredPathToGenerateHtmlReport}
- Once the above command completes execution, ensure that the HTML report gets generated at the specified folder location.
Let’s open the generated report and see the different sections. The important sections depicted by the report are highlighted below.
As you can see, the above report is far more easy to read and interpret and is useful for all stakeholders of the project to get a glimpse of the overall test execution summary.
Executing Selective Tests
Very often, you are required to execute a few selective tests or a subset of the entire test suite or tests marked with some special categories. All this is possible via Specflow tags that we saw in our previous articles (which can be applied both at Feature level as well as Scenario level).
Thus, in order to execute just a subset of tests, we can use Specflow tags to mark/label such scenarios and while running through the NUnit console runner the categories can be mentioned as a part of the command.
Let’s see the step by step process to perform selective test execution:
#1) Open the Visual Studio Project and let’s tag a Scenario/Feature which we want to run selectively with a tag/category named as “selective”.
#2) Compile the solution for the changes to take place and a new DLL with updated changes gets created.
#3) Now open the command prompt and follow the steps as mentioned above for executing the tests with NUnit console application.
#4) The NUnit executable looks at the test filter and tries to search for all the tests matching the specified filter. In our case, it will try to match all the feature/scenarios with category “selective” and execute only those tests.
Since we just tagged one scenario with this tag, we should expect the NUnit test runner to execute just one test.
nunit3-console.exe "--result={PathToReportFile};format=nunit2" --where "cat == selective" {PathToSpecflowSeleniumProjectDll}
As you can see above, we’ve mentioned filter like “cat == selective”. The NUnit application interprets it as a category filter with the value “selective”.
#5) Once the above command execution completes, we can see the test summary on the console (as well as the generated XML report by NUnit).
#6) Now, we can create an HTML report out of it (by following the steps to generate HTML through Specflow executable). This is how the generated HTML will look like.
As NUnit just executed one test, the generated HTML report should show only one feature and one scenario.
Let’s see how the HTML report looks like:
Conclusion
In this tutorial, we explored the different ways of executing Specflow feature tests along with generating HTML reports through the Specflow executable. The generated reports are easy to read & interpret and are useful for all the stakeholders of the project.
We also saw the techniques to execute a subset of tests rather than the entire test suite through Specflow tags which enables NUnit to look/filter for specific scenarios and execute only those which match the specified filter criteria.
Check out our upcoming tutorial to explore the Most Frequently Asked Interview Questions on Specflow!