Visual Regression Testing with Nightwatch.js & Applitools

Nightwatch.js is an automated testing framework for web applications and websites, written in Node.js. It is a complete browser testing solution which aims to simplify the process of setting up Continuous Integration and writing automated tests (source: http://nightwatchjs.org)

In case you are not familiar with Nightwatch.js, you can follow this simple guide on how to get started quickly: http://nightwatchjs.org/gettingstarted

Adding Visual Regression Tests to Nightwatch.js

Once you have everything installed, up and running and you were able to run your first test with Nightwatch.js you are ready to add Eyes into the test.

I’m using the “nightwatch/examples/tests/google/googleDemoTest.js” as my codebase for this example.

In the test below, I navigate to a website, grab a screenshot and send it to Eyes – this image is now my baseline. I run the test again at a later point and compare with my baseline. Now the content has changed which means that a simple bitmap comparison is essentially useless. We will need Eyes AI Powered engine to process the screenshots and alert us if a layout change is found, meaning a bug exists on that page.

var Eyes = require('eyes.images').Eyes;
var fs = require('fs');

module.exports = {
  'grab screenshot' : function (client) {

      client
        .url('https://medium.com')
        .waitForElementPresent('body', 1000)
        .saveScreenshot('./homepage.png')
        .end();
  },

  'send screenshot to Eyes' : function(client) {

  var eyes = new Eyes();
    eyes.setApiKey(process.env.APPLITOOLS_API_KEY);
    eyes.setMatchLevel('Layout');

    var image1 = fs.readFileSync('./homepage.png');

  var firstTestPromise = eyes.open("image comparison", "medium", {width: 800, height: 600}).then(function () { 
     return eyes.checkImage(image1, "homepage");
  }).then(function () {
     return eyes.close(false);
  }, function (err) {
     console.error("An error occurs during runtime", err);
     return eyes.abortIfNotClosed();
  });
  client.end();
  }
};

Note that I’m setting Eyes Match Level to Layout, this means Eyes will ignore the content and will only check the layout of the page to see if it was not broken (to read more about Eyes Match Levels check out this article). If we had used Exact or Strict match levels both would fail as there are significant differences between the two screenshots (see screenshots below)

Baseline

Checkpoint

Analyze the Test Results

Since everything has changed on the page, images & text, It’s hard to compare and point out the bugs at first glance but Eyes spots it instantly. See the test results in our dashboard below.

Test Results

In Applitools dashboard we can quickly spot the mismatch between our baseline and checkpoint. When we zoom in we can actually see that the word “vice” in the checkpoint is cropped compared to the baseline where the text appeared as it should be.

Conclusion

If you’re not using Visual Testing yet I encourage you to start today, functional testing is critical for your app however it does not replace a human looking at your page to check for scenarios you did not think of when running your scripts. And when considering adding visual testing to your test framework make sure you choose the right tool that can help you identify the issues quickly and reliably, don’t settle for simple bitmap comparison as it will not work in the test case above.