Pact - Contract Testing

Owned by Jeremy Bowman (Deactivated)
Last updated: Aug 22, 2023
4 min read

Overview

Pact is the most widely used open source tool for a software testing method called consumer-driven contract testing. It’s a way for consumers of an API defined elsewhere to declare their expectations of that API, have those expectations enforced in the producer’s test suite, and make that all work with a minimum of manual effort. This is much more efficient than end to end tests for detecting and preventing code changes which would result in different services (or the front and back end code of the same service) becoming unable to communicate with each other as intended. As such, it can be a very useful tool for creating performant test suites for microservice architectures and micro frontends that actually catch most potential issues that would cause errors in the production environment.

History and Commercial Support

Originally developed in 2013 at the Australian contract development firm DiUS to address pain points in the Ruby on Rails microservice architecture of a real estate site. The proprietary PactFlow extension to the core open source Pact Broker was released in 2019 to provide a revenue stream for the core developers to work on Pact full time. In 2022, the PactFlow team was acquired by SmartBear, a company headquartered in Somerville, MA maintaining several other quality assurance tools (some open source, some proprietary). The SmartBear acquisition has somewhat improved the range of commercial support options and documentation for the tool.

If you want more details:

Programming Language Support

The original implementation was in Ruby (for a system of Rails microservices). The framework soon started expanding to other programming languages: Java in 2014, JavaScript in 2016, Python in 2017, etc. The reference implementation of Pact is now implemented in Rust, with most of the other supported languages transitioning from being independent implementations to wrappers around this reference core. JavaScript has already switched over, the Python switch is still in progress (https://github.com/pactflow/roadmap/issues/92, https://github.com/pact-foundation/pact-python/issues/88). The Python implementation only supports the older V2 Pact specification instead of the latest V4 until this is complete.

Current Open edX Usage

Pact is already used for interactions between https://github.com/openedx/edx-val and 2U’s proprietary video encoding manger; the public side is implemented in https://github.com/openedx/edx-val/blob/master/.github/workflows/ci.yml, and 2U pays for the PactFlow service as the contract broker which keeps all the code related to the contracts properly synchronized.

There is also a usage of Pact for testing communications between frontend-app-learner and edx-platform, in this case not utilizing PactFlow. The implementation of this was added in https://github.com/openedx/edx-platform/pull/28289 and https://github.com/openedx/frontend-app-learning/pull/510 .

Conference Presentation

Dawoud Sheraz played a leading role in the Open edX pilots of Pact and recently gave a presentation on Pact at Python Web Conf 2023:

History in Open edX

Pact was discovered by some edX staff by late 2017 and advocated as a potential solution for recurring pain around the performance, efficacy, and maintenance of end to end tests. But it wasn’t until 2021 that a pilot project to use it was actually approved. The implementing team at Arbisoft presented the results, but there wasn’t a strong followup (the learning materials available at the time didn’t lend themselves to quickly grasping when and how to use the tool in new services). In 2023, it is again being promoted as a potential solution for some common types of deployment incidents, especially as microservices and micro frontends proliferate in the Open edX system.

Creating Pact Consumer Tests

The first step in creating a Pact between a consumer and provider is generating a contract on the consumer side. This contract will encompass the entirety of expected requests and return values between the two services. This includes paths of API endpoints, HTTP status codes, and response bodies of these API calls. A contract can contain as many interactions as deemed fit to ensure validity between the consumer and provider. The Pact Foundation package will produce this contract, and thus we’re given flexibility in the testing framework we want to implement in our codebase. These contracts will be fed into a general Pact stub server ran on a Docker container with a specialized image, detailed here. These contracts will be of further use to us when we integrate tests into our provider service.

Creating Pact Provider Verification Tests

After setting up our consumer tests, we need a way to verify our contract against our provider service. Fortunately, Pact has an in-built Verifier than can match up a instantiated Provider state with what our contract expects of it. We simulate our Consumer request to the Provider, and we alter our Provider state in regards to what should be returned when a certain endpoint is hit. It is important that we discard and clean up all of our testing states after our Pact Verification has concluded. This slideshow gives a visual example of the verification system that is used on Pact consumer-generated contracts.

Next Steps

We need to find and/or create good resources for developers to quickly learn how and when to use Pact. Usage will remain limited until we can get many developers to view contract tests as just another standard quality assurance tool to use alongside other kinds of automated tests when appropriate. Potential resources will be collected in a child page for review and curation.

Related Tools