This guide is for manually running Visual Regression tests on local dev environment. For Automated Visual Regression testing, check the relevant guide.
In order to catch css regression bugs before they land on production, or ideally before even merged, we use use a tool (BackstopJS) for comparing visual changes, test regressions and verify changes.
The way this tool works is that it creates a series of screenshots for predefined urls and device resolutions and it automatically recognizes visual differences caused by our css code changes and reports them back in a way we can easily inspect them on browser and decide if these are within accepted threshold or not.
First thing is to make sure we have all required dependencies in our local repository. The toolset for these tests live inside the master-theme repo. So using either npm we can simply install all the things we need:
You should also make sure you have gulp installed. It’s the tool we use as a runner for our js scripts. If your operating system or distribution doesn’t provide a package you can use npm:
sudo npm install -g gulp
Before you start coding a new frontend feature you should create reference screenshots:
gulp backstop_reference --url=http://example.org
At any time you can create test screenshots:
gulp backstop_test --url=http://example.org
Once this is finished a report will be launched in your browser in order to inspect the visual diff.
On the browser report you can quickly see which views have some diff and filter out the ones that look identical, by clicking on the “Failed” button. These will also have a diff screenshot.
By clicking on that diff, or at the “Show Diffs” button, will get you to the visual inspection tool. Besides looking at the reference and the test screenshot, you can inspect the visual differences in two ways. Either with the “Diff” tool, which paints out the differences, or with the “Scrubber” tool, which provides a more interactive way.
The screenshots below showcase these 3 steps.
If the url option is omitted the tests will run against the stage environment. Ideally you should this option to point to your local docker planet4 hostname.
You can even use different urls for reference and test. For instance to compare local environment with production. But this may catch some false positives due to potential content differences.