Visual Regression Tests

Catching frontend regression bugs

In order to catch css regression bugs before they land on production, or ideally before even merged, we use use a tool (BackstopJS) for comparing visual changes, test regressions and verify changes.

The way this tool works is that it creates a series of screenshots for predefined urls and device resolutions and it automatically recognizes visual differences caused by our css code changes and reports them back in a way we can easily inspect them on browser and decide if these are within accepted threshold or not.

Automated Visual Regression testing

By default, for all Planet 4 websites, when their staging website is being build/deployed, the following happens:

1. Before the deployment, a screenshot of certain elements is taken

2. After the deployment a new screenshot is taken

3. A report is generated to see if the two screenshots match.

When a backstopjs job fails on staging, the workflow triggers a "Hold" workflow for the developers to visually look at the report, and decide if they want to proceed or not.

If the backstopJS job does not fail the workflow triggers directly the production deploy pipeline.

How to see the report

1. Go to the CircleCI workflows for your website

2. Get in the latest production workflow.

3. Find the job visualtests-compare and get into it (click it so that you go in the details)

visual tests job

• Go to the tab artifacts.

• Drill down to find the folder html_report

• Click on index.html. It will open in a new tab the report with the comparisons

backstop find report

If you are happy with result go back to the workflow and approve the hold-production job.

hold approve

Manual Visual Regression testing

Developers can run the regression tests locally as they work on a feature branch.

You can either mount the same docker image to your local docker environment and modify the json file, or you can go full hardcore and install and configure BackstopJS locally. There is a simple backstop configuration already inside the master-theme to get started.

Backstop infra code

The test get run by a custom Planet 4 docker image that includes all the necessary dependencies for backstopJS and the scripts relevant to the above workflow.

The docker image is generated by the repository planet4-backstop and the docker images are stored in docker hub.

In the readme file of the above repository you can read how you can do modifications and local testing.

The default page and viewport settings are controlled by the file backstop.json in the above repository.