Automated Tests Browser Compatibility

Owned by Jeremy Bowman (Deactivated)
Apr 03, 2023
5 min read

We have a few categories of test cases which rely on browser automation to verify that our software is working as intended:

  • bok-choy acceptance tests
  • lettuce acceptance tests
  • karma/jasmine JavaScript tests
  • pa11ycrawler for accessibility verification

These were set up using browser versions that were convenient to install and use at the time, but we haven't been great at updating to newer versions or making sure that things work across different browsers.  We're currently getting most of these working in a new Docker-based Devstack where we can isolate the browser used into a separate, independently upgradeable container, so this is a good time to take stock of where we are at the moment and what constraints are limiting us from just using the latest version of whatever browser we want.

Vagrant Devstack

The Vagrant Devstack currently comes with Firefox 45.0.2 (for bok-choy and karma tests), Chrome 55.0.2883.87 (for lettuce tests), and PhantomJS 1.9.8 (for pa11ycrawler).  All of our tests currently work with these versions because they're what the tests were written with.  However, there are a few good reasons why we don't want to just keep using these indefinitely:

  • They aren't representative of the browsers our real users are utilizing (Firefox users are in the minority, and are more likely to be on newer versions like 54).
  • PhantomJS is no longer being maintained, the former maintainer having decided that the advent of headless Chrome makes it redundant enough to no longer be worth working on.
  • The software we use to perform browser automation (like Selenium and karma) is gradually dropping support for these older browsers.  We need to keep upgrading these packages for bugfixes and useful new features, which will eventually force us to replace these browsers.

Due to the advent of Docker Devstack (see below), we may not actually upgrade the browsers installed in Vagrant.  But it's important to know that we shouldn't plan to just copy this existing configuration exactly for new environments.

Jenkins

We have some Jenkins worker nodes which have been upgraded to use Ubuntu 16.04 and currently have the following browsers installed (matching Vagrant):

  • Firefox 45.0.2
  • PhantomJS 1.9.8
  • Chrome 55.0.2883.87

There are still some older worker nodes which are still using Ubuntu 14.04, and have correspondingly older browser versions installed:

  • Firefox 42.0
  • PhantomJS 1.9.8
  • Chrome 30.0.1599.114

Docker Devstack

At the moment, Docker Devstack uses an old copy of one of the official Selenium Docker images, chosen in order to be close to the version being used in Vagrant.  The chosen image has Firefox 47.0.1, but does not currently pass all tests for either bok-choy or karma (it was committed once the vast majority of the tests were passing).  These are the problems which have been identified:

  • The image does not include the codec libraries needed for mp4 playback, so many of the video-related tests fail.  This can be resolved by creating our own image which just adds one layer to the official image, adding the missing libraries.  There is a well-maintained 3rd-party Docker image which has these pre-installed, but it also comes with a lot of other functionality which makes it much larger than the official Selenium images.
  • Because Firefox is being launched by a remote Selenium server instead of using a binary local to the container running the test code, it runs a little differently.  The main problem identified with this to date is that the launched Firefox window doesn't have focus from the operating system's perspective, so a number of focus-related tests fail.  This can be fixed by enabling the "focusmanager.testmode" Firefox preference through the Selenium driver, but this doesn't work correctly through the JavaScript Selenium driver until fairly recent versions of the official Docker image (using Firefox > 47.0.1).
  • The lms JavaScript tests are failing to run due to a relatively new Webpack build issue which is also currently affecting Vagrant.

We're running Firefox in a separate container for a few reasons:

  • It lets us significantly reduce the size of the edxapp container, which changes much more frequently.
  • It allows us to leverage all the work being done by the Selenium community to provide working, reliable images with VNC connectivity and support for a variety of browser versions.
  • It allows us to run browser automation tests from any of our services without needing to add the required software to each service we want to test in.
  • We can run different browser versions in different containers, if necessary; it's often very difficult to install multiple versions of Chrome or Firefox in the same container or virtual machine.

Lettuce tests are not yet working under Docker Devstack, they were overlooked in the first pass at getting all the tests running.  But adding a Chrome Selenium container has proven straightforward, so adding support for this shouldn't take too long.

Upgrade Options

Here are the consequences of trying to upgrade and/or consolidate the browsers used for each test suite:

Firefox 54

  • The JavaScript tests fully pass (except for the unrelated lms issue mentioned above) with the latest Firefox running via a remote Selenium server, as long as the codecs for mp4 playback are installed.
  • There is at least one open bug in controlling Firefox versions newer than 47.0.1 via a remote Selenium server which prevents many of our bok-choy tests from passing.
  • An older version of Firefox was deemed unacceptably slow for pa11ycrawler; this may have improved somewhat in recent versions, but PhantomJS is probably still significantly faster.

This should be fine for the JavaScript tests, but nothing else yet.

Chrome 59

  • Exactly one JavaScript test fails when run with the latest Chrome: the test for seeking to a specific location in the video player.  Chrome only allows this when the server supports the HTTP "Accept-Ranges" header, which the Django test server does not yet support.  We can probably skip this test when using Chrome, but should be wary of only using Chrome in CI if we do that.
  • bok-choy: still investigating; hitting some early problems that may just be local environment issues.

Chrome 59 (headless)

Chrome 59 added support for fully headless operation, which should allow for faster test performance and smaller installations which don't need xvfb and such.

  • There isn't yet an official Selenium Docker image that supports this, although it's being worked on, and there is an unofficial image from that work which is supposedly mostly functioning.
  • There's a bug in taking screenshots which isn't fixed until Chrome 60 (not officially released yet).
  • This loses the ability to view and interact with a running test suite via VNC like we can with the regular Chrome and Firefox debug containers.  That doesn't prevent us from using it when we aren't actively exercising that ability, though.
  • It will probably suffer from the same JS video seek test failure as regular Chrome.

This looks like it may be a good option in a few weeks, once all the kinks get worked out.

PhantomJS 2.1

This upgrade hasn't been attempted yet (and may not be worth the effort, given that the project has been abandoned and we probably need to migrate pa11ycrawler to a different browser anyway).