How to automate a visual regression test

One of the biggest nuisances when releasing a product or feature are regressions in your codebase. Some functionality that was working just fine two weeks ago is now broken – and it's always the client who discovers the problem. Without automated testing, you're doomed to either:

1. Spend too much time manually testing functionality for every change
2. Run a greater risk of unknowingly breaking existing functionality

Despite this knowledge, many of us don't bother writing automated regression tests, as we feel it's just too much work to set up and maintain. Luckily that's changed. We can now write fairly comprehensive tests using a technique called visual regression testing.

Regressions occur when functionality stops working as expected. Usually this happens after a code change, but it can also occur due to changes in page content or time-based conditions (e.g. daylight savings messing up alarm clocks).

We test by comparing the look of a website before and after changes (i.e. diffing). This is not test-driven development, where you write your tests and then your code – instead, you' re testing against the visual output itself.

Getting started

To keep things simple, I'm testing a basic login screen. I forked a neat little Plunkr written by Jason Watmore. I'll be testing on a local copy, but I could easily test using the original Plunkr itself.

There are a plethora of testing tools available and each have strengths and weaknesses. For example, Wraith and BackstopJS are great for quickly testing a large swathe of pages. You simply add the URLs of the pages you want to check along with CSS selectors to focus on and run it.

This works extremely well for static sites, but won't capture the complexity of modern web apps. In those cases, you'll need to dive into test scripting. This means writing a set of actions to take – essentially, teaching the computer to browse your website. CasperJS is great for this, but doesn't allow you to test across the entire browser suite.

In order to capture the full scope of testing we're aiming for, we're going to rely on a tool the industry has been using for years: Selenium.

Installing Selenium

In the past, using Selenium has meant installing a Java runtime and writing a bunch of Java code for the test cases. Thankfully for frontend developers, that's changed.

Setting up Selenium locally is now simple for anyone familiar with npm. Thanks to the selenium-standalone module, all I need to do is run three commands in my command line.

npm install -g selenium-standalone
selenium-standalone install
selenium-standalone start

This gives me programmatic access to the browsers installed on my computer, allowing me to use WebdriverIO, an npm module that provides Selenium bindings for Node.js. I'll be using a related tool called WebdriverCSS. WebdriverCSS is essentially an add-on to WebdriverIO, adding on visual regression testing.

To install both, I run npm install webdriverio webdrivercss from the command line of my project root folder.

Writing my test

Now I have everything installed and Selenium running, it's time to write my first test. To keep things organised, I created a 'tests' folder, and inside it a subfolder named 'visual'. This folder will house both our tests and our screenshots. I then created an empty JavaScript file named 'login.js' .

The first thing I need to do is load WebdriverIO and WebdriverCSS. This is a Node.js script, so I' ll be using its built-in require statements:

var webdriverio = require('webdriverio');
var webdrivercss = require('webdrivercss');

Next I initialise WebdriverIO, and define which browser to test in:

var client = webdriverio.remote({
desiredCapabilities: {
browserName: 'firefox'
}
});

By passing in 'firefox' as the value for the browserName property, I tell WebdriverIO to use my local copy of Firefox. The desiredCapabilities object can contain a lot more information, especially when testing on a more advanced Selenium set-up. However, for my needs, this is all that' s required.

There's one more item I have to take care of before starting on my actual test case. I need to initialise WebdriverCSS and define a couple of paths for storing images.

webdrivercss.init(client, {
screenshotRoot: 'tests/visual/baseline',
failedComparisonsRoot: 'tests/visual/failures',
});

You can change the paths to match your setup, or rename baseline and failures to something different if you prefer. The main point is to keep it all inside the ' tests/visual/' folder.

Basic website connection

Now that everything is set up, it's time to write the actual tests. To do this, I first specify the website I'm testing:

client
.init()
.url('http://localhost:3000')
.end();

There's also some initialisation and cleanup code surrounding the URL definition, which indicates to WebdriverIO when it should start and stop the test.

Define screenshot areas

Now comes the actual test. I'm going to capture two areas of the login form: the page header and the form itself. To define this, I call the webdrivercss function and pass in my details.

client
.init()
.url('http://localhost:3000')
.webdrivercss('login form', [{
name: 'title',
elem: '.container h2'
}, {
name: 'form',
elem: '.container form'
}])
.end();

login form, title and form are all human-friendly names used to generate the image filename of the screenshot. The other part is the elem property. This is what Selenium will use to find the element you're looking for.

While it's common to use a CSS selector for this, you also have the full suite of selector strategies available via WebdriverIO.

Comparing changes

Now we have the test script written, it's time to try it out. I open the command line back up and execute node tests/visual/login.js from my project root. I wait a minute or two for it to complete the execution. During this time, a Firefox window pops up with the website being tested, and I can watch as the tests run.

After the tests have completed, I'm returned to my command prompt. Initially, it looks like nothing happened, but when I open my 'tests/visual/baseline' folder, I now have three new images in it: 'login form.form.baseline.png', 'login form.title.baseline.png' and 'login form.png'. Our main focus is the '.baseline' images. The third image is more of a reference of the overall page being tested (in case I need it for debugging).

Note: If you're running your tests on a Retina display, your screenshots will likely be off. This is a known issue with WebdriverCSS. The best workaround is to use an external Selenium Grid provider like BrowserStack or Sauce Labs to run your tests.

Catching regressions

The baseline images don't tell me much, apart from the fact that they've captured the right area of the page. To see regressions, I have to introduce a change and run the tests again.

I'm going to change the title of my page to 'Login to My Website' and run the tests a second time. Now that I have baseline images available, WebdriverCSS is smart enough to know to compare the latest images to the baseline.

Once my test is complete, two new images are now available. The first, inside the 'baseline' folder, is named 'login form.title.regression.png' and shows the newly captured screenshot. The '.form' image remains the same, as no changes occurred between the two test runs.

The second image ('login form.title.diff.png' ) is in the 'failures' folder and is the diff output. This image is generated by ImageMagick and highlights the differences between the '.baseline' and '.regression' shots. This helps us quickly grasp what parts of the image have actually changed.

Accepting change

If you've made changes on purpose, you'll want to update your baseline. To do this, replace the old '.baseline' image with the new '.regression' one by deleting and renaming the files. To help with this process, the WebdriverIO folks created an Admin Panel interface that really improves the workflow. Check it out.

Final thoughts

While visual regression testing is still a new discipline for frontend, usage is steadily growing. It works with the visual nature of the web to offer a relatively easy way to gather useful functional regression results. Invest the time to try it out and you'll have more confidence next time you update your codebase.

Words: Kevin Lamping

Kevin Lamping is a senior frontend engineer at InVision. This article was originally published in issue 272 of net magazine.

Liked this? Read these!

• 4 tips for a smarter CSS workflow
• Useful and inspiring flyer templates
• Download free textures: high resolution and ready to use now