Contract-driven testing is an agreement between API consumer and provider, written as tests, on how an interface behaves. The provider runs those tests in CI before deployment, and the same tests generate a mock the consumer can develop and test against. In a large bank this decouples system testing from fragile integration environments, where one broken feed stalls every team.
Key Takeaways
- Contract-driven testing turns the agreement between API consumer and provider into tests the provider must pass in CI before anything reaches the integration environment.
- Mocks built from a developer’s imagination of an interface are not realistic, which is why 100 percent code coverage can sit next to terrible code.
- The same contract tests generate a mock, so consumers and testers work in their development environment instead of waiting for 50 integrated applications.
- Phil Royston sold contract testing to senior management with one promise, shorter testing in the release cycle, while middle managers holding the budgets took longer to win over.
- Approved business requirements turned out not to be testable; it took testers asking why and a painful round with the analysts before the bank had requirements worth testing.
A Core Banking System That Nobody Can Test Until Fifty Other Systems Are Up
Contract-driven testing is what Phil Royston landed on when a large Czech bank asked him to help with a transformation program and he found that no team could test anything on its own. The core banking system had once been a standalone monolith. A tester could run system tests against it in isolation. Today the same system does nothing unless it is integrated into around 50 other applications, and on any given day one of them is down. The LDAP service is broken, the exchange rate feed is broken, the client data feed is broken, so the whole environment is stuck, and with it every tester who depends on it.
The bank is about 100 years old. In 35 years it went from paper-based processes under communism to a smartphone app that is supposed to feel like Revolut, with retail, corporate, investment and private banking under one roof, several merged banks in its history and millions of clients. Production quality is high, and the people keep it that way through a lot of testing. The price is that every change ships in big releases, because everything is wired to everything else. DevOps has preached small batch sizes for 15 years. A bank in this position cannot do small batches. To go live with one change, ten wheels have to turn at once.
That was the starting point for Phil. He had done a test process assessment at the bank a year earlier and seen the structural problems. When the transformation program started, he saw the one moment in which some of them might actually get fixed.
What Is Contract-Driven Testing and How Does It Work between Consumer and Provider?
Contract-driven testing starts with an agreement between the consumer of an API and its provider on how the interface behaves, written down as tests. Most people hear “contract” and think of field definitions: this attribute is mandatory, that one is numeric. The contract can carry far more than that. It can hold use cases. If the consumer calls the account service with this specific account number, the answer is “account does not exist”; with that other number, the answer is a valid account. The expected behaviour becomes a set of tests, and the set of tests becomes the contract.
From there the approach works like test-driven development across a team boundary. The developer of the provider has to run the contract tests against the API implementation before integration, as part of CI, before the build or deployment goes anywhere. Passing them proves one thing: the provider has not broken what the consumer expects from the interface.
For the bank this fixes a problem that caused years of pain. Teams changed interfaces, deployed into the acceptance testing environment where everyone else was trying to test, and ten systems stopped working. With the contract tests in the pipeline, far less broken integration reaches that shared environment in the first place.
A Mock Generated from the Contract Behaves like the Real Provider
Testing in isolation always needs mocks, and that is where unit tests and isolated system tests have a weak spot. The mock is based on the programmer’s idea of how the other component will behave. It is not real, it is imagination, and tests against imagination are not reliable. Phil sees the same symptom everywhere: developers report 100 percent code coverage and the code is still bad.
“When developers say they’ve got 100 percent code coverage and the code is still terrible, there’s something wrong.”
(Phil Royston)
The various shapes that have replaced the classic test pyramid, the Christmas tree, the honeycomb, the trophy from Spotify and others, all point the same way: unit testing on its own is not very effective, so test the interfaces, because that is where the errors live. Those interfaces are exactly what contract testing pins down.
Here the contract pays a second time. From the agreed tests, a mock or simulator can be generated that behaves exactly as the tests describe. The consumer team develops against this mock and gets realistic results, maybe not the final ones, but much closer to reality than anything they have today. The provider tests the API on a local machine. The tester tests against the mock in the development environment and can cover a good part of the functionality there. Testing is decoupled from the integration environment, and the acceptance environment stops being the place where everything breaks for the first time.
Phil’s goal for the next one to two years is to see this decoupled system testing in action: a system under test with every neighbour mocked out, built from contracts that were agreed before development started. Not in six months. The organisation is big and slow to change, and every step needs many conversations.
How Do You Win Over a Bank That Has Tested the Integrated Way for Twenty Years?
Phil’s first step, only half a joke: know how to write a PowerPoint slide. Senior management in a bank carries a KPI to release more often, and they are looking for a solution. Before pitching one, talk to them. Find out what keeps them awake at night, which production incidents hurt, what those incidents do to their work. These people operate at a high level because that is all their day allows; they cannot sit through 20 slides on the technicalities of contract-driven testing. What they need is one message: here is contract testing, this is what it enables, and it should shorten the testing phase in every release cycle. The answer was “approved, go for it”.
The harder part came after that. Convincing senior management is often the easy bit; convincing everybody else is the work. Phil describes himself as the high-level guy who understands, sells and explains the theory, and he was open about not being able to do the next step himself. That step belonged to a couple of strong technical people who dug into the details, showed code snippets and ran demos. Once the engineers in the room saw a demo, they were on board.
Between the two levels sits the group with the most power to veto: middle managers, because they own the budgets. They need neither a strategy slide nor a code snippet. So Phil coaches his technical people to explain the goal and the concept first, since their habit is to jump straight into the code. In a large organisation, communication on all three levels decides more than the tooling does. And when the change gets hard and people want to give up, somebody has to get them back on the train.
Approved Business Requirements Are Not Testable Business Requirements
A second change at the bank ran under the label business-driven development. A senior business stakeholder said that his people had no time for testing: project teams turned up demanding user acceptance testing right now, while the business was busy with something else. Phil offered help on one condition. The testers had to understand what the business needed and expected, and little of that was written down; most of it lived in people’s heads. Since the bank was implementing a new system and banks write business requirements anyway, the team agreed to use those.
Then the pattern showed up. A business person tells a business analyst what she wants. The analyst writes it down accurately and reads it back, and the business person confirms it, because that is what she said. A tester reads the same document and asks why. Why do you want that, what is this for, how would we find out whether it worked? That was a painful learning curve for the business analysts and solution architects: an approved business requirement is not a testable business requirement. Today the bank has testable requirements, and the whole business analyst community of practice supports the process, because everyone accepted one argument. You cannot expect testers to know whether something works if nobody can tell them what it is supposed to do.
Two more changes have taken hold alongside it. The teams use the test pyramid more deliberately and have moved a lot of UI testing down to API testing. For end-to-end testing, which in a bank means processes spanning around 50 applications, Phil did not want yet another central team doing end-to-end on top of what every squad does. Instead, the squads borrow each other’s tests to cover the full chain, an approach the team calls “parasitic testing”. The test managers have adopted it because it saves them time and energy, even if it does not yet work at organisation level.
Why Does a Testing Transformation Need Measurement from Day One?
The learning Phil puts first after 18 months is the one he missed: he did not start measuring. When a change program runs, people are busy doing things, and none of it is visible. This does not have to mean a metrics dashboard. At minimum it means checking regularly what is happening, what is in place and what is still missing, and feeding that back to the people involved.
What works for him now are interviews with the test managers. Ask them directly what they are doing about contract tests, about API testing, about requirements. The interview is also a communication moment. Most test managers have heard about the program, few have understood it, and nearly all are too busy to act on it. In a conversation you can connect the program to their actual problem: if we tried this on your project now, would it help you? Phil counts too little communication and too little feedback as the two gaps that slowed the first year and a half.
Wait for the Crisis, Then Move
Nothing in this program moved fast, and Phil did not expect it to. His advice to anyone in a similar position is to make small changes, move things forward, expect no miracles and wait for the crisis. He used the same tactic as a test manager for years. He would join a project where the business had approved the specifications, read them and see that nobody could seriously have approved them. The project kept running towards its milestones. Telling everyone at that point that signed-off is not approved changes nothing. The moment arrives when the problems start, and that is when the question “why don’t we do it this way?” gets an answer.
At the bank the crisis has a name, the transformation program, and the fact that an outsider is walking around who looks like he knows what he is talking about is part of what gets people engaged. Some are curious and want in. Others see the unknown ahead and want someone next to them. Both are reasons to stay patient and take the opportunity when it comes.
