The product ecosystem at LinkedIn is vast, and managing its infrastructure can be daunting. Nearly 10 years ago, we transitioned away from a single monolithic codebase toward a microservice architecture, allowing teams to manage their own repositories. Today, we have thousands of developers working on more than ten thousand active repositories, some of which have upwards of 300 commits a day flowing through them.
We practice trunk-based development, in which developers of a given repository push changes to a main development branch (e.g., “master”), rebase frequently, and avoid long-lived feature branches. In this blog, we will focus on how our Continuous Integration (CI) system is able to work with repositories of different sizes, specifically ones with a high velocity of commits being merged into master, to ensure timeliness and code correctness.
In any code management system (especially when trunk-based development is used) there are two main constraints to consider in relation to developer productivity. The first constraint is that master should aim to always be “green.” This means that at any point in time, if the codebase is checked out at HEAD (the latest revision), it should be able to compile and build successfully, as well as generate any necessary artifacts. When master becomes “red” (i.e., it is unable to build successfully at HEAD), it requires either a revert back to the last known good revision or for an engineer to step in to make a new change that fixes the problem and allows the product to build again, often referred to as “fixing forward.” Both of these options can potentially take a significant amount of time, causing delays for any newly added features or fixes that were added between the bad change and fix.
The second constraint is that developers expect their changes to land in the repository in a timely manner. The simplest approach to merging multiple changes to a single codebase is to queue up every change that is being submitted and then, one at a time, perform validation/testing and merge them into the repository. The biggest drawback to chronologically merging changes like this is that it simply does not scale for high-velocity repositories. Consider a product that takes an hour or so to build, or one that requires a large amount of validation work before a change can be accepted. If there are multiple engineers each pushing multiple commits a day to this repository, the queue of changes will continually grow and things will quickly spiral out of control. Another approach is to batch commits together and if any of them fail, reduce the size of the batch and try again. However, this too runs into issues at scale when you have, say, 20 commits in a single batch, as a single bad commit could cause unnecessary reruns.
In addition to the previous two, a third constraint exists for us due to our usage of a microservice-based architecture; the system should remain performant for both high- and low-velocity repositories. It is important that the system doesn’t optimize only for high-velocity repositories and cause developers of low-velocity repositories to suffer productivity losses. We solve this by allowing developers to customize some of the validations that they wish to run throughout the pipeline (as well as when they wish to run them), as we will discuss in the next section.
LinkedIn uses both pre-receive (pre-merge) validations as well as post-receive (post-merge) validations in order to ensure the satisfaction of the constraints that we have laid out above—this is the norm of many trunk-based development CI systems. Once a developer is ready to push their change, they use an internally-developed git sub-command called “git submit” rather than running “git push.” This CLI will kick off a validation job and immediately return control back to the developer. As depicted below, the job will then run pre-merge validations and do a “git push” on the user’s behalf, rebasing as necessary. Once complete, the code will be merged into the repository and post-merge validations will be kicked off through a server-side post-receive hook.