Contract testing - Who really needs it?

Consumer-driven contract testing is an approach to API testing in which the consumer of an interface specifies what data it sends and what responses it expects. These tests result in a machine-readable contract against which both sides test independently of each other. If a test fails, the software cannot be built.

Key Takeaways

• Consumer-driven contract testing solves a structural problem: the consumer defines what data and formats they expect from an interface, not the provider.
• The repository-based approach with Open API and Renovate only checks the syntax and structure of an interface, but does not recognize semantic changes such as the removal of individual enum values.
• Pact generates a machine-readable contract from consumer tests, against which both sides can test independently of each other without requiring the remote site to be live.
• If a contract test fails, this blocks the build: The software cannot be released as long as the contract is broken.

Three ways to secure an API against changes

Anyone offering or consuming a REST interface has roughly three options for keeping changes under control. The simplest is just talking: Two teams sit down together, clarify the interface contract verbally and are done quickly. From a testing perspective, this is rarely enough.

The other two methods are technically supported and can be automated. The first is repository-based and versions the interface description itself. The second is contract testing, in which the consumer formulates their expectations of the interface and makes them testable.

Both approaches are sensibly based on Open API. The interface is first described with Open API and then generated from it. This saves programming effort and is less error-prone than writing the interface by hand.

How the repository-based approach with Open API and Renovate works

With the repository-based approach, the Open API description is treated as a separate, versioned artifact. In a Java environment, the generated interface ends up in a Maven repository that only contains this interface description.

The second component is Renovate. The tool reads the Maven POM file and compares the dependencies in it: not only those of the software, but also those of the plugins. If Renovate detects a new version, it can automatically create a merge request in the connected repository.

If the build of this merge request runs, Renovate can also merge it automatically. If the build fails, the merge request is left for a manual check. Someone then checks whether there is a breaking change and what needs to be adjusted.

The benefit is that an interface change no longer slips through unnoticed. The consumer receives a new version, has to adopt it and stumbles across what has changed. This also works if it is not known who is using the interface because the producer simply publishes the change.

Why the repository approach only checks structure and syntax

Renovate recognizes changes to structure and syntax, but not semantic shifts. This is the central limitation of the approach.

An example makes this clear. A parameter is defined as a string, but the consumer actually only processes two permissible values, i.e. a kind of enum. If the producer deletes one of these values, the syntax remains unchanged: It is still a string. This change is therefore not noticeable via the repository-based comparison.

A principle from the interface design applies here. What an interface delivers should be described as strictly as possible. What comes in should be interpreted benevolently and as much as possible should be allowed through. As long as syntax is involved, this can be described well. As soon as semantics are involved, a different kind of test is needed.

Consumer-driven contract testing: the consumer defines what they expect

In consumer-driven contract testing, the person who wants to use the interface describes what they expect from it. They define what they send and what they must receive in response. This also allows semantics to be tested.

Andrej Thiele uses the Pact tool with its builder pattern for this. Using a DSL, the consumer describes in detail that they are sending a GET request with certain data and receiving data back in a defined format with certain variations. He writes a separate case for each variation that interests him.

This solves the enum problem from the repository approach. If the consumer expects exactly two values, they specify both as examples. If these values are suddenly omitted, the test fails immediately. The consumer only tests what he actually evaluates. If he is interested in three of many fields, he only writes tests for these three values. He doesn’t care about the rest.

This is how the workflow between consumer and producer works

The process begins on the consumer side. The test is written there first. This test and its data are used to create a contract, a document that is either stored in the file system or on a Pact Broker.

The Pact Broker can be secured via HTTPS so that only participants who know the interface and want to comply with the contract can access it. Both consumer and producer read the same contract.

The framework generates a kind of mock server from the contract. On the consumer side, the software fires its request directly against it. On the producer side, exactly the data that the consumer has specified is sent from the contract to the producer, so that the producer is also checked with real expected values.

This decouples the collaboration. The consumer does not have to wait for the finished third-party system, but already works against the contract. Several consumers can write their own tests independently of each other, which do not have to overlap in terms of content.

Contract testing does not work without talking

Contract testing does not replace coordination between teams, it just makes it less necessary. It is usually set up so that the software cannot even be released if one of the contract tests fails.

This is exactly what forces a discussion when in doubt. If one team changes something and another does not follow suit, the person affected will see that something has failed, but will not necessarily know exactly what has happened. Then they have to talk to the other team.

Even as a computer scientist, you have to talk terribly. Andrej Thiele

The hard advantage of this mechanism: nobody can avoid the test. If the test doesn’t run, the software doesn’t build.

When which approach is worthwhile

Contract testing originates from the microservice environment. There, the interfaces are deliberately kept small, so there are no masses of tests. In more complex interfaces with five or six different consumers, each writing their own tests, the test basis can grow significantly.

The following comparison summarizes the differences:

A mixed solution is possible. For example, you start with the Renovate approach and add contract testing where semantics are important. The expansion is iterative: not every consumer has to join in from the start, the setup grows bit by bit.

What you need for a setup

The effort required depends on your existing technical know-how. For a demonstration setup, the entire chain can be recreated locally in a Docker environment, with your own GitLab, a Nexus in the Community Edition and a Renovate server. Renovate can also run as a GitLab runner.

Pact testing is the easier way to get started. You don’t necessarily need a Pact Broker for this, the exchange of contracts also works via the file system or a shared directory that both sides can access. The Renovate variant is much easier to handle during operation.

For integration and reporting, everything remains within the usual framework. Pact tests are written as normal JUnit tests and run as integration tests. The output comes from the JUnit framework and can be reused in the pipeline. The Pact Broker also shows when contracts have been broken. In essence, however, you can see the result directly in the software: it does not build if the test fails.