This guide is for Automated Visual Regression tests. For running manually Visual Regression tests on local dev environment check the relevant guide.


How this works

To allow for quick identification of changes that alter a website’s appearance we have added in the release process visual regression testing.

We have used the technology BackstopJS .

By default, for all planet4 websites, when their develop and staging/release website is being build/deployed, the following happens:

For the develop websites, a non match does not stop the process, just appears in the circleCI workflow as “failed” (red). Example:

But the process continues normally, and the release/staging website rebuild is triggered normally.

For the release/staging website though, there are differences.

When a backstopjs job fails on staging, the workflow triggers a “Hold” workflow for the developers to visually look at the report, and decide if they want to proceed or not.

If the backstopJS job does not fail, then under certain conditions (aka, the text [AUTO-PROCEED] to exist in the git commit message), the workflow triggers directly the production site rebuild/redeploy. This brings us a step closer to continuous deployment.

How to see the report

How to approve a job that is on hold

To accomodate from the different scenarios (hold or no hold) we had to seperate the original “release” workflow into three different workflows :

The release-init runs always. If according the criteria it decides to put things on hold for approval, it will trigger release-hold-and-finish. Else it will trigger release-finish.

To approve a job that is on hold, you have to

How to add extra tests on your own website

Add a file called backstop-pages.json with a key scenarios. The format and available options can be seen at the backstopjs documentation. The simplest form of the file should include a label and a URL for each page, like the following:

  "scenarios": [
      "label": "Planet4 KoyanSync Explore page",
      "url": "https://APP_HOSTNAME/APP_HOSTPATH/explore/"

How you can test different sizes:
Currently the sizes are hardcoded at the file (in this repository) backstop.json:

width: 320, height: 480
width: 1024, height: 768
You can add extra view ports there, but they will be used in all Planet4 websites.

If you want to add extra viewports to one site only, you can do it via the backstop-pages.json by adding something like that:

  "viewports": [
      "label": "widescreen",
      "width": 1366,
      "height": 768

Please be very careful, as any viewport or page you add multiplies the number of screenshot that are taken, and things could get out of hand easily. (For example, the default has 1 page X 2 viewports, and takes 2 screenshots. By adding the above 1 viewport and 2 pages, we have in total 3 X 3 = 9 screenshots)

For local dev environment:

You can either mount the same docker image to your local docker environment and modify the json file, or you can go full hardcore and install and configure BackstopJS locally

For Planet4 devs/infra:

The test get run by a custom planet4 docker image that includes all the necessary dependencies for backstopJS and the scripts relevant to the above workflow.

The docker image is generated by the repository planet4-backstop and the docker images are stored in docker hub .

In the readme file of the above repository you can read how you can do modifications and local testing.

The default page and viewport settings are controlled by the file backstop.json in the above repository.

Extra notes: