This Comprehensive Guide Explains All About Visual Regression Testing. Also includes Review of Free & Commercial Visual Testing Tools like Applitools, Percy, Screener, etc:
Visual Regression testing or Visual testing in software is a quality assurance activity of verifying if the visual aspects of the application’s user interface seem appropriate to the user. It is also known as Visual Validation Testing.
It aims at verifying if the correct data and content are getting displayed at the application’s front end. Additionally, it also validates the layout and appearance of each visual element present on the user interface and the whole UI itself.
What You Will Learn:
- Visual Regression Testing
- List Of The Top Visual Testing Tools
Visual Regression Testing
When we talk about the correctness of Layout, it means that positioning, shape, and size of each element on the screen is correct. Elements should not overlap with each other, they should be visible and not hidden.
The visual element here may refer to an image, page, part of a page or part of an image.
Validating appearance means that the font, color, brightness, contrast, density, texture, visual weight, graphics, etc. of the visual elements must look proper.
What Is Visual Regression Testing?
Regression testing ensures that any change introduced to the software does not break anything that was previously doing fine.
When it comes to Visual testing, the visual regression testing ensures that any style issues don’t pop up, when any changes are done in the software. The application should continue to look as good as it was.
A visual regression testing tool captures screenshots of the current UI and compares them with the original screenshots. So, visual regression testing looks at the historical differences. It confirms that the web page still renders as expected across various browsers even after modifications to the code.
Given below is a screen-print of a Visual Regression Test run taken through an automated tool:
In today’s practices of continuous integration, visual regression testing becomes very important to ensure that the new changes do not cause any mess in the layout as the application develops from version to version and progresses across browsers.
Many organizations employ visual regression testing to record versions of certain web pages to document how it changes from one iteration to another.
Importance Of Visual Validation Testing
No matter how correctly the functionality of your application is working, it won’t be of use to the customer unless it can provide a good user interface and user experience.
These days, we not only talk about UI (User Interface), but we also focus a lot on UX (user experience). When we are targeting to deliver better user experience, then visual testing becomes much important because the more appealing visual design can be perceived as more usable to the user.
Let’s see the below example from the Amazon website:
In the above image, you can notice that the text & images are overlapping, and they are not properly aligned. This is an example of a visual bug that can create a bad user experience as the customer won’t be able to complete the intended action on the website.
Visual design shapes the user experience in many ways.
- Great visual designs help to builds trust and credibility.
- Reinforces brand image.
- Affects legibility.
- Forms a visual balance on the screen.
- Guides the users towards actions.
- Persuades the eye to shift its attention towards certain page elements.
Also, nowadays we have a huge variety of operating systems, web browsers, screen resolutions, and devices, thus it becomes crucial to ensure that the UI is appearing correct on each of the possible combinations of these.
Even a small distortion in the UI can result in loss of business. Visual bugs can be annoying and unpleasant to the users and create inconvenience for them.
Suppose you are into the banking business and you offer an online mobile banking application to your customers.
You get complaints from some of your customers that when they want to transfer money to someone’s account and after they fill up all the required information, they are not able to find the ‘Submit’ button on the screen and this is an irritating experience for them.
When you investigate the issue, you find that the submit button gets hidden on a particular screen resolution for android mobiles and that is why all the customers with Android phones of that particular screen size were not able to see the submit button.
So, the application’s functionality was working fine in this case but because of the visual bug, the users were not able to complete their intended actions on the app. This example shows how important it is to test the visual aspects of the application on every possible configuration before you deliver the application to the customers.
Some areas where visual validation testing is highly recommended are (because it will directly impact your brand):
- Mobile Apps
- Mobile Web/Responsive Web
- Marketing websites
- Content management systems
- Consumer systems like airlines, travel, banking, etc.
Given its importance, companies should spend some significant effort and money on Visual testing. If used in combination with the existing developer tests, it will help you in detecting the visual issues in the early stages of the development lifecycle.
Visual Testing Methods And Processes
As developers or testers, we often work on testing the functionality of the applications. Either through manual or some automated tool, we check if the application is working as expected. Such functionality testing is very important to ensure the smooth working of the application.
However, if we add visual testing to this, then it would improve the overall testing significantly. Thus, this testing is often done in combination with functional testing.
There are several testing methods and processes to meet the challenges of visual testing of applications over wide configurations. There are also a lot of tools available to support visual validation testing.
Generally, there are two main approaches followed for visual validation testing i.e. Code-based, and Configuration based. The visual testing tools are also categorized based on these two approaches.
It can be done manually or in an automated manner. Due to certain challenges involved, it can neither be fully manual or fully automated. Generally, a combination of manual and automated approach is preferred.
Given below are the 3 different Visual Testing Methods that are generally followed.
Teams can choose between any of these options depending upon their needs:
#1) Create Dedicated Visual Tests: This is the most recommended method as it gives you full control over visual validation. Also, using this approach, you can write tests with moderate efforts. But yes, you will have to write the tests from scratch and it will need some significant time.
#2) Insert Visual Checkpoints in Existing Functional Tests: In this method, you already have the tests for validating the functionality of the application and you just add some visual checks between these tests to validate the pages.
Of course, this method limits your test coverage but it’s a quick way of inserting visual tests and take the benefit from the existing functional testing code.
#3) Insert Implicit Visual Validation to the Existing Testing Framework: This method is very easy to implement as it requires only a few lines of code. However, it suffers from the limitation that you can also do generic validation in this case. You can’t validate specific components in the framework.
As we move on, we will discuss Manual and Automated visual testing. We will also check the visual testing tools and frameworks in detail.
Manual Visual Testing
Software companies are faced with two options to perform visual testing i.e. Manual testing or Automated testing. Let’s see Manual visual testing in detail.
Although manual testing can be slow, expensive and error-prone, due to the subjective nature of the interface design, it is often not possible to have an automated test to capture the correctness of every scenario. So, at times we need to choose a balance between Manual and Automation testing.
Also, one of the major advantages of manual testing by a human being can look outside the scope of the test. An automation test can only capture the issue that is in its scope, but in a manual test, a human tester can also spot out other cosmetic bugs and other things that look incorrect on the UI.
Since manual tests are closer to real-life user actions, they can capture the layout bugs and design issues that are generally uncovered when a real user interacts with the application.
You can go for manual testing at the early stages of the application development when your UI is unstable. It is also helpful for ad-hoc testing when you need to have quick on the spot checkups.
In manual visual validation testing, the tester has to manually check the appearance and layout of the visual elements. This may involve taking current screenshots and then manually comparing them against the baseline screenshots to identify any gaps.
Another effective way to do manual testing is to use a component explorer. In this, you define the test states, use explorer to select a state & component and check it on the screen. One such tool is React Storybook where you can develop UI components in isolation.
Through this method, a tester can run a manual test of a component, check how it renders and see if it is working fine or not.
Manual Testing Steps:
- Browse through the app and capture the known good baseline screenshots.
- At a later point, take another set of screenshots from the latest version of the app.
- Now, manually compare these screenshots (baseline vs current) to figure out the visual bugs.
Automated Visual Testing
Automated visual testing is a way to automatically verify if a user interface visually appears as intended.
The benefits of automating visual tests are that they offer long term cost efficiency, are faster than manual tests, more accurate as they can exclude the human errors & deliver pixel-perfect visual tests, they are reusable & transparent as they offer automatic reports that are easily and readily accessible by anyone in the team.
Some of the limitations of automated visual testing are that there can be large upfront expenses and you will need to spend significant efforts in test maintenance to accommodate every change.
In automation testing, the scope is narrower unless screenshot testing is in place. Also, there is a steep learning curve as the organizations take time to learn about automation testing tools.
To go with automation would be a good choice if you are required to perform regression visual testing to deal with frequent changes happening to a stable UI. Automated testing also helps in a great visual screenshot comparison.
Automated screenshot comparison offers a great degree of precision in visual testing and increases the ROI. The automated screenshot comparison can capture those bugs that are impossible to get detected with human eyes and manual comparison. It is also helpful in the end to end testing for complex user stories.
There are a wide variety of open source and commercial tools available for automated testing.
The below image shows how visual validation can be automated using any programming language:
Given below is the general workflow of Visual Test Automation:
As shown in the above workflow diagram, there are 4 steps involved in Automated Visual Testing:
Step 1: In this step, we need to drive the application & the test and capture screenshots.
Step 2: In this step, the automation tool compares these screenshots against the baseline screenshots. The baseline screenshots are generally the images that were taken in the previous test runs and proven by a tester.
Step 3: Once the tool obtains the results of the image comparisons, it generates a report that highlights all the differences found.
For example, the below image shows one of the test automation tools highlighting the differences in an angular application:
Step 4: In the last step, the tester reviews the report and checks for each difference as to whether it’s a bug or a valid change (false positives). Based on this, the baseline images are updated.
For the first test run, you don’t have baseline images. So, the images obtained in the first run are generally treated themselves as baseline images. From the next run onwards, they are compared against the screenshots.
Best Practices For Automated Visual Testing
- Pick a tool that is smart enough to ignore false positives. The tool should be able to handle anti-aliasing, pixel offsets, etc. and does not fail your tests because of these reasons.
- Test automation should be able to handle dynamic and moving content.
- Don’t rely on error ratios or configuration of thresholds. The only thing that should matter is whether a human being can see that difference, and will that impact the user experience?
- The automation algorithm should be able to analyze the structure of the page and should be capable of doing layout comparisons.
- Prefer Validating full UI page instead of individual components. This will provide more and better coverage. In the case of validating only specific components, you can miss unexpected bugs.
Automated Testing Tools And Visual Testing Frameworks
Today, there is a wide variety (almost 30 plus) of tools available in the market for automated visual testing. Some of these tools are open source while some are commercial. Most of these tools function in the same manner, but they pursue a different workflow as per the target audience.
For front-end developers, there are two types of frameworks involved in visual testing. First, you require a test runner that allows you to write and execute the tests. Second, you require a browser automation framework to replicate user interactions. These two frameworks are jointly called as Visual test code.
With the help of these testing frameworks, the developers create the code that imitates real user actions like typing text or clicking buttons. The test code has commands to take screenshots at relevant points. When the test is executed for the first time, an initial set of screenshots is taken.
These screenshots act as a baseline for comparison when any changes made in the application. Once the baseline is founded, the developer executes the code in the background. When a change is found, the screenshot of the change is captured.
The test runner compares this screenshot against the baseline screenshot for that area of code. If any differences are found between the images, then the test is considered a failure and the visual bug is reported.
When the whole test code is executed, a report is generated automatically. This report is manually reviewed by a human tester to check all the screenshot images that are different from the baseline images. Some tools generate a diff image to highlight the difference between the actual and expected screenshot.
If the difference is caused by an issue in the code, then the developer can fix that issue and rerun the code. If the difference is caused by a required change in the UI of the application, then the developer reviews that screenshot and updates the baseline screenshot so that the future tests can be passed.
Visual Testing includes both free and paid tools. Let’s discuss the tools in detail.
List Of The Top Visual Testing Tools
Code/Script Based Open-source & Free Tools
Enlisted below is the list of the code-based visual validation tools that are open-source and available at a free of cost.
- PhantomCSS (UJs CapserJs)
- FBSnapshotTestCase (Uses XCTest)
- Gemini (Uses JS DSL)
- Needle (Uses Python)
- Rspec Page Regression (Uses Capibara)
- Pix-Diff (Uses JS and Protractor)
- Selenium Visual Diff (Uses Java and WD)
- Vizregress (Uses .NET and WD)
- VisualCeption (Uses PHP and CodeCeption)
- Specter (Uses a JS DSL)
Configuration Based Open-source & Free Tools
Configuration-based visual validation tools that are open-source and available free of cost are listed below for your reference:
- dpxdtMake –Depicted
- Grunt Photobox
- CSS Visual Test
- Snap And Compare
- Galen Framework
We will not go deep into each of the above tools, however, let’s discuss in brief about the best ones.
Phantom CSS is one of the very popular open-source tools for visual validation. It has 4,768 GitHub stars. It aids you with automated visual regression testing. It uses CasperJS to take the screenshots and compares them with the baseline screenshots using Resemble.js.
It generates the image diffs based on RGB pixel differences. This tool is useful only when the UI is predictable.
FBSnapshotTestCase is also another popular tool in this category. It has 880 GitHub stars.
It captures a configured UIView or CALayer and employs a method called render InContext: to acquire an image snapshot of its contents. It matches this image against a “reference image” kept in your source code repository and fails the test in case the two snapshots don’t match.
BackstopJS is a leading tool in the category of configuration based tools. It catches the CSS Curveballs. It is an automation tool for visual regression testing that can test the responsive Web UI by matching the DOM screenshots over time.
It is an easy to use tool with some great features like in-browser reporting UI, integrated docker rendering, JUnit reports, CLI reports, etc.
Galen framework is also one of the famous open-source frameworks for visual validation testing. Using this tool you can test the layout and responsive design of web apps in an automated manner.
This framework runs well in the selenium grid.
Let’s discuss some commercial tools and frameworks available for Visual Test Automation now!
It is one of the very popular commercial frameworks for automated visual testing and visual regression testing. This tool provides an AI-powered cognitive vision. It uses artificial intelligence to aid you with end to end visual testing and monitoring.
With Applitools, you can automatically test hundreds of UI components across all platforms and configurations without writing any code or with very minimal code. It has support for more than 40 testing frameworks and languages.
It also supports the DevOps process where you can easily integrate your visual test cases into the CI/CD pipeline. This tool also lets you create custom visual reports.
This framework is being employed by some Top companies including Sony, SAP, MasterCard, and PayPal. It has very positive customer reviews too.
The vendor offers three versions of this framework i.e. Starter, Enterprise Public Cloud and Enterprise Dedicated Cloud. You will need to contact the vendor for the pricing.
#2) Crossbrowsertesting Visual Testing
Crossbrowsertesting is a web testing platform developed by SmartBear for the entire testing process. This tool covers visual testing as well as visual regression testing.
The screenshot testing feature provided by this tool captures full-page screenshots and quickly helps in finding visual bugs and tracking visual inconsistencies. To start a screenshot test, you need to provide the URL and pick some browsers.
Screenshot tests are available for most of the desktop devices and popular mobile devices. You can mix and match your browser resolution for dynamic responsive testing across desktop browsers and mobile orientations. Your screenshot test will capture three views of each configuration i.e. Windowed, Full-page and Full-page chromeless.
You can quickly filter your test results and see the visual bugs. You can also jump to a live test and debug or correct the visual defects. You can test local and development environments through a local connection tool provided by this software.
For visual regression testing, once you have successfully executed a screenshot test, you can just schedule it on a daily, weekly or monthly basis. The tool will send out the test result notifications too.
This tool has an automated comparison engine to capture automated screenshots of the same page across various configurations. From here, you can select a baseline browser, and then evaluate the highlighted layout differences alongside.
It also offers some advanced options to make your test more powerful. These options include Basic authentication, Login profile, Selenium script, Screenshot delay, Send emails, Hide fixed elements, etc.
This is a priced tool. However, a free trial is also available. The Live testing version of this tool will cost you $29/month, Automated testing is priced at $60/month and Unlimited testing is priced at $100/month.
#3) Percy Visual Testing
Percy is one of the powerful tools for automated visual testing. It allows you to integrate, execute and review the visual tests. Integration can be done via test automation frameworks, CI/CD services, or straight through your application.
After integration, we can start executing the visual tests required on the applications and components. When a visual test is started, the Percy tool extracts UI screenshots throughout various browsers and responsive widths. It then does a pixel by pixel baseline comparison and identifies any relevant visual changes in your UI.
Once the screenshots are rendered, you can review them for any visual issues.
The pixel by pixel diffs and responsive diffs provided by this tool gives great visual coverage. Also, the snapshot stabilization feature minimizes false positives.
This is a paid tool. However, it offers a free trial version too. The paid versions have three flavors i.e. Essential, Business, and Enterprise. The Essential version is priced at $29/month, the Business version is priced at $849/month and the Enterprise version is custom created as per your need, hence its price will vary accordingly.
Website: Percy Visual Testing
This is an automated testing tool that lets you record and execute the tests in realtime in the cloud. Test flows can be easily automated with no coding required. It automatically spots the UI inconsistencies across different platforms. It also supports the storybook component testing.
It allows you to do visual testing in combination with functional testing in the same test run which thereby increases the overall test coverage. Companies using screener.io include Yammer, Microsoft, uber, etc.
The Starter pack of this tool will cost you around $249/mo, the Perform pack is priced at $499/mo and the Enterprise pack has custom pricing based on your requirements.
endtest is a UI testing Platform that supports codeless automation testing with the help of machine learning. It allows you to quickly create the automated tests, store and execute them directly in the cloud. They also have a chrome extension through which you can record the tests.
This tool has a lot of features like generate random test data, advanced assertion, automatic backups, geolocation, live videos, screenshot comparison, etc.
In this tutorial, we discussed how important visual validation testing is in today’s scenario where the software application is required to be working fine on multiple configurations of screen sizes, mobile devices, operating systems, screen resolutions, etc.
If you want a good user experience, then visual validation testing is as important as functional testing. Visual testing and Functional testing, when combined can provide wide test coverage.
There are a wide variety of open-source and commercial tools & frameworks available for visual validation testing. We discussed some of the best tools in this tutorial. Those tools can indeed help you with Automated Visual Testing and Visual Regression Testing.