In the microservices architecture, API becomes the boundary and bridges between many components. It defines a clear contract for interaction between service providers and consumers. We observed several challenges regarding microservices testing.
1. Breaking API Changes
In the microservices architecture, making API changes is a norm. Unfortunately, we’ve already seen a few production incidents caused by breaking API changes. A few quotes from past incident postmortems:
“Adding a new data field broke the listing availability service’s API xxx before the corresponding service code changes got deployed. The problematic deploy got automatically rolled back after the errors shot up.”
“Schema field deprecation should only happen after we are sure the fields are no longer used. However, a change to remove fields that are used in xxx was deployed first with no gem upgrade on the xxx side later. The backward incompatible change broke xxx service.”
2. Lack of Usable API Mock Data for Service Consumer
API is the hotspot for Test Doubles ( mocks, fakes, stubs, etc… of production objects) across various places like unit tests and integration tests. Without support, engineers are duplicating time and effort to create mock data, clients, and services repeatedly in different places.
It also leads to engineers experiencing unnecessarily heavy weight test setup. For example, spinning up dependent service and even dummy databases. Those heavy efforts are counter-productive. Tests are supposed to be light and frequently run.
3. Lack of Assurance for Mock’s Semantic Correctness
In addition to the upfront cost, it is difficult to catch the semantic correctness of mocking data because service business logic is constantly evolving, and the manually maintained test data easily becomes out-of-sync. This results in tests becoming less useful overtime.
4. Lack of Validation for Service Owner
API is the boundary between a service and its consumers. It’s a minimum requirement for API to be implemented as expected.
However, without framework level validation support, service owners are finding their own heavy lifting to set up validations against their API endpoints, which can be different from the set of validations used by the API consumers.
5. Lack of Real Time Metrics for API Test Quality
Last but not least, engineers have almost zero insight into API test coverage and test data quality. For API tests, there is no metric equivalent to code coverage, which is a widely used metric that helps measure the quality of unit tests.
6. Most Importantly: Test in a Lightweight and Scalable Manner
Encouraging engineers to build tests in a monolithic style is the last thing we want. Such hidden monoliths will bring us back to square one: slow tests, untrustworthy test results, tight coupling, and breaking a terrifying number of tests with small changes.
To tackle the above challenges in a lightweight and scalable manner, we find that service schema is the keystone.
Note that, there are so many critical components needed to make a successful testing story, organization-wise education and best practices, a scalable test runner, continuous integration infrastructure(CI), continuous delivery infrastructure(CD), testing environments, etc. In this post, we will be only focusing on the schema aspect of service API testing.