This Pact Contract Testing tutorial explains what is Consumer-Driven Contract Testing, how does it work and why should you use it in your testing strategy:
What is Contract Testing?
Consumer-Driven Contract Testing is a form of API testing which truly enables shift left. The contract tool we use is Pact.io, and we will learn about it later in this series of tutorials.
Contract testing is a method to verify integration between two applications independently in order to test what has been passed and see if what is returned matches with the “contract”.
Contract tests fit nicely within a microservice architecture, operating in an agile setting. Therefore examples will be based on the experience that we have gained while working in this environment.
What You Will Learn:
List Of Tutorials In This Contract Testing Series
Tutorial #1: Introduction to Contract Testing With Examples [This Tutorial]
Tutorial #2: How To Write A Consumer Pact Test In JavaScript
Tutorial #3: How To Publish Pact Contract To Pact Broker
Tutorial #4: Verify Pact Contract And Continuous Deployment With Pact CLI
Consumer-Driven Contract Testing
The starting point is your API documentation which forms the contract for your tests, at this point usually, the development teams take the API document and develop against the wiki document (or whichever format it resides in your organization, such as Word Document).
For example, a Web Application where the front-end is being developed by Team Krypton and the API is being developed by Team Thoron. The project starts with a kick-off meeting where the requirements are presented and agreed upon between the teams.
Each team takes the requirements and starts creating the backlog by refining stories. The development starts in both teams following the user stories, integration testing is left for later sprints. As Team Krypton finds additional requirements, relating to error scenarios the API documentation is updated accordingly.
Test cases are added by Team Thoron related to the updated scenarios based on the documentation.
Already we can see a couple of flaws with this process, and I’ve added a couple more for good luck:
- API document changes may not be communicated effectively.
- Front-end team stubs out back-end service and vice versa.
- Back-end team creates integration test cases based on documentation.
- Integration environment is the first time when full integration is tested.
- Different API version on integration environment vs production.
Consumer-driven contract testing has two sides i.e. the consumer and the provider. This is where traditional thinking about testing in microservices is flipped around.
The Consumer is the curator of the scenarios, including the request and the expected response. This allows you to follow Postel’s Law which dictates you should be flexible in what your API can accept but conservative in what is sent. Referring back to flaws no. 1, 3, and 4, the documentation changes are driven by the consumer.
For example, in the circumstance where Team Thoron changes a string field to not accept null values, the consumer tests would not reflect the change and therefore would fail. Or at least until the changes had been made on Team Krypton.
[image source]
The Provider verifies the scenarios provided by the consumer against their “dev” environment. This allows your microservices to enforce Parallel Change which states that you should expand the API functionality, followed by migrating to a new version. Referring back to flaw no. 2, the stubs usually created by the back-end teams for their own testing requirements can now be based on the consumer scenarios using Pact Stub Server.
The binding element of the two sides is the “contract” which needs to be shared between the teams. The pact provides a platform to enable the sharing of contracts called the Pact Broker (available as a managed service with Pactflow.io).
The Broker stores the output of the consumer scenarios. The contract is then stored within the broker alongside the version of the API. This enables testing against multiple versions of the API, thus compatibility can be confirmed before release, as highlighted in flaw no.5.
An added benefit to the Pact Broker in the legacy platforms is the visibility of consumers. Not all consumers have been known to the API authors, especially it’s not how it is being consumed.
Specifically referring to an occurrence where two API versions were being supported, there was a data issue within version 1 (V1) whereby the API was causing dirty data in the database.
The change was implemented in V1 of the API and pushed to production, however, the consumer relied on the format which was causing the data issue, thereby, breaking their integration with the API.
How Does It Work
The example above shows the authentication flow, the web service requires the users to authenticate in order to access sensitive data. The web service sends a request to the API to generate a token using a username and password. The API returns a bearer token which is added to the data request as an authentication header.
The Consumer test constructs a POST request for a token by passing the body with username and password.
A mock server is spun up during the test which validates the request you construct, along with the expected response which in this example includes the value for the token.
The output of the consumer test generates a pact contract file. This will be stored in the pact broker as version 1.
The provider then pulls version 1 from the pact broker and replays this request against their local environment, by verifying the request and response match with the consumer requirements.
Roles And Responsibilities
Quality Assurance (QA) / Tester: Creating contracts using Pact.io and working with the BA to generate the test scenarios.
Developer: Pairing with the QA’s on creating the tests and helping wrap the API for implementing in Continuous Integration (CI).
Business Analyst (BA): Generating the scenarios and working with the architect to verify affected parties.
Solution Architect (May not exist in your organization): Actioning the API changes and co-ordinating with the BA on implementation, also communicating changes to consumers (using the Pact Broker to understand whom it may concern).
Release Management: (Yes I know it’s old-fashioned, but still exists in my world): Filled with confidence that changes will be released successfully due to contract testing coverage.
Whole Team: Verify the results to determine whether the releases can be pushed to production with the Pact CLI tool, Can I Deploy.
Contract Testing Vs Integration Testing
Integration testing has to exist in order to validate if the system is working before promotion to the production environment, but the scenarios can be significantly reduced.
The impact of this could be:
- Faster feedback before releasing to the integration environment.
- Less reliance on the stability of the integration environment.
- Fewer environments supporting multiple API versions.
- Reduced unstable environment instances due to integration issues.
| Integration | Contract | |
|---|---|---|
| API Configuration | Yes | No |
| Deployment Checks | Yes | No |
| API Versioning | Yes | Yes |
| Debug Locally | No | Yes |
| Environmental Issues | Yes | No |
| Feedback Time | Slow | Fast |
| Clearly Pinpoint Failure | Many layers | Very Easy |
Firstly, contract testing does not replace integration testing. But it probably can replace some of your existing integration test scenarios, shift left, and provides faster feedback to your software development lifecycle.
In integration testing, you will be verifying the context in which the API lives, such as the environment architecture, the deployment process, etc.
Therefore you want to be running the core test scenarios which would confirm the configuration, for example, the health check endpoint for the api version. Also proving whether the deployment was successful by returning a 200 response.
In contract testing, you are testing the specifics of the API, which includes the edge cases related to the API structure, content (E.g. field values, keys exist), and error responses. For example, does the API handle null values or are they stripped from the API response (another real example).
Some Benefits (If you’re not already sold)
Enlisted below are some of the benefits to draw upon while selling contract testing to the wider business:
- Faster deployment of software
- A single source of truth
- Visibility of all consumers
- Ease of testing against different API versions.
Frequently Asked Questions
Some common questions while trying to persuade people to adopt contract testing include:
Q #1) We have 100% test coverage already so we don’t need it.
Answer: Well that’s impossible, but contract testing has many other benefits than just test coverage.
Q #2) It’s the Solution Architect’s responsibility to communicate API changes.
Answer: Quality is the whole team’s responsibility.
Q #3) Why are we creating the test scenarios for the API team?
Answer: The API team doesn’t know how the web service works, so why should it be there responsibility.
Q #4) Our end-to-end tests cover the whole flow from start to finish, including other integration points.
Answer: Exactly why we are splitting the tests to test one thing and it’s not your responsibility to test the end-to-end flow of a system that you don’t know how it works.
Q #5) In which team’s repository does the tests live?
Answer: Both. The consumer in their repository and Provider in theirs. Then in the central point, the contract lives outside of either of them.
Arguments
These are the arguments that we find it hard to argue against when it comes to transitioning to contract to test:
- Swagger documentation already in place which can be used to generate integration tests.
- Teams own both front-end and back-end services with an effective mechanism for API changes.
Continuous Integration
How does this fit into your continuous integration test suite? The desirable place for contract testing to live is with your unit tests.
Consumer tests spin up a mock server that requires no external dependencies outside of the test.
Provider tests require an API instance, therefore the local API can be wrapped using an in-memory test server. However, if it is not easy to wrap your API locally, a workaround that we have used previously is where we spun up an environment and deploy the code to this environment as a part of the pull request automated checks.
[image source]
Conclusion
In this tutorial, we learned what contract testing means and what it looks like in a microservice infrastructure, and saw how it looks in a real-world example.
Lessons have been learned about how contract testing can help you shift your integration testing to the left. In addition, we saw how it can reduce costs to your organization by reducing feedback times related to integration issues.
Contract testing is not only a tool for technical testing, it enforces the collaboration of development teams by communicating changes and encouraging testing as one unit. Overall, this should be a prerequisite to anyone looking to move to Continuous Deployment.