Pinot is architected in a simple way that makes mass migrations relatively easy and pain-free. There is a single-source-of-truth stored in a central location—such as a Network File System, Azure Data Lake, or another type of object storage. Pinot servers begin downloading any missing data segments that they are assigned, and brokers begin routing queries to the servers as soon as they finish downloading and processing segments. This means that the servers themselves are relatively easy to replace.
The migration approach
The general approach for migrating Pinot servers is simple: Spin up a new set of machines to migrate to, update the assignment policy to point to the new machines, and trigger a cluster rebalance to start moving segments around. The new machines will begin replicating and serving queries from the assigned segments. That means the old servers can be turned off to complete the migration. Brokers are even easier because they are entirely stateless. However, there are some special nuances to our upgrade that require additional care.
The upgrade to RHEL7
Pinot is a general real-time analytics database, and shines at providing up-to-date answers and statistics for a wide variety of our member-facing features. All this information is updated and accessible on-demand, which means that there are strict latency requirements to help deliver positive member experiences. Pinot provides a simple rebalancer that, for most use cases, is quick and effective, but sometimes can have a noticeable on performance. To secure us from these latter instances, we needed to find a different method.
Pinot is also host to a lot of our data at LinkedIn. Replicating all of this data back down from the single source of truth to all of the servers (for each of the replicas) would greatly slow down the migration. During this window of time of waiting for a new server to finish replicating its copy of the segments, the cluster is operating under reduced performance and resilience capacity. It was imperative that we found a way to shorten this time.
Instead of using the standard Pinot rebalancer, we instead devised a custom approach to migrate each cluster. We built up an identical destination cluster, and then prepared a mapping of each source server to a destination server. With these servers paired up, we performed parallel rsyncs from each source server to its matching destination to stage the data.