Initial problem

We noticed that there was a version mismatch in a PHP library. Since it’s a dependency of a dependency, we didn’t have a specific version defined in our Composer requirements. The problem was triggered as the latest version of that library, that was automatically pulled, introduced a new function. At the same time, some plugins were installing an older version of that library, causing PHP to crash in certain scenarios.

Mitigation

To resolve the issue, we pinned a specific version of that library that satisfies all plugins’ requirements. That effectively solved the problem. The problem never reached production since it was identified already on development sites and in our local development environment.

Regression on affected sites

Some sites started reporting the same issue after a production deployment. Looking into logs, it was easy to determine that the problem was caused by this library version mismatch. The reason for this regression is that those websites are pinned in an older P4 release. This was done since there was a child theme incompatibility that required more time to be fixed (eg. Timber 2.x upgrade).

Lessons

  • Websites shouldn’t stay on an older P4 version for too long. Pinning an older version is an acceptable workaround when a P4 release requires further work for a specific child theme, but it should be done as a temporary measure.
  • Some of the affected websites are using the Yoast SEO plugin, which is one of the plugins pulling a different version of that PHP library. Our fix covers that case as well, but it also highlights why installing unsupported plugins requires more testing also from the website developers.
  • On a positive note, our systematic usage of Grafana and Sentry for debugging these types of issues allowed us to pinpoint the exact problem relatively quickly and apply a fix with confidence. Both of those tools are also available for NRO developers.
  • Cloudflare full page cache helped minimizing the incident’s impact on end users. The issue primarily affected logged-in users.