The anomaly detector identifies two types of anomalies:
Broker failures: i.e., a non-empty broker leaves a cluster, which results in under-replicated partitions. Since this can happen during normal cluster bounces as well, the anomaly detector provides a configurable grace period before it triggers the notifier and fixes the cluster.
Goal violations: i.e., an optimization goal is violated. If self-healing is enabled, Cruise Control will proactively attempt to address the goal violation by automatically analyzing the workload, and executing optimization proposals.
The executor is responsible for carrying out the optimization proposals from the analyzer. Rebalancing a Kafka cluster usually involves partition reassignment. The executor ensures that the execution is resource-aware and does not overwhelm any broker. The partition reassignment could also be a long-running process—it may take days to finish in a large Kafka cluster. Sometimes, users may want to stop the ongoing partition reassignment. The executor is designed in a way that it is safe to interrupt when executing the proposals.
We’ve encountered many interesting challenges while developing and using Cruise Control. A few of these are listed below.
Building a reliable cluster workload model for Kafka
This is not as simple as it sounds. There are quite a few nuances to be aware of. For example, it is straightforward to collect CPU utilization metrics from brokers, but how do we quantify the impact of each partition on CPU utilization? This wiki page explains our efforts to answer this question.
How long are you willing to wait for an optimization proposal?
The analyzer component has come a long way. We initially used a general-purpose optimizer with a complicated parameterized loss function. It would take weeks (if not years) to get the optimization proposals on a medium-size Kafka cluster. We then switched to the current heuristic optimizer solution, which gives us a reasonably good result in a few minutes.
Memory or speed?
Cruise Control is very memory-intensive due to the quantity of metrics that we have to keep for a period (e.g. a week) in order to profile the traffic pattern of the partitions in the Kafka cluster. It is also a CPU-intensive application because of the involved computation that is necessary to generate optimization proposals. Those two things are sort of conflicting with each other, however. To accelerate proposal generation, we would want to do more caching and parallel proposal computation, but doing that uses more memory. We ultimately made some design decisions to strike a balance between these two. For example, we pre-compute the optimization proposals and cache them to avoid long waiting times when user queries arrive. On the other hand, we also stagger the execution of the memory-heavy tasks (e.g., proposal precomputing, anomaly detection, etc.) to avoid simultaneous high-memory consumption.
More Kafka cluster optimization goals!
Since the optimization goals of Cruise Control are pluggable, users may come up with sophisticated and nuanced goals to optimize their own Kafka clusters as needed. For example, we use Kafka Monitor at LinkedIn to monitor our cluster availability. Since Kafka Monitor reports availability for each broker based on its ability to send messages to a “monitor” topic, we need to ensure that the leaders of this topic’s partitions have coverage across all brokers. As an open source project, we would also like to encourage users to create their own goals and contribute them to the community.
Integration with Cloud Management Systems (CMS)
Currently, Cruise Control heals a cluster by moving partitions off dead brokers. We envision that Cruise Control could integrate with other CMS to automatically expand clusters when their utilization reaches a certain threshold, or to replace a dead broker with a new one from a spare pool if necessary. As noted above, we welcome community input and contributions to this future functionality.
Empower operations with insights
Cruise Control facilitates deep analysis on metrics collected from Kafka. We believe it will equip SREs with the ability to quantify the impact of various resource usage metrics and derive insights that will aid in capacity planning and performance tuning.
We developed Cruise Control with the realization that a dynamic load balancer is a useful facility for any distributed system. Cruise Control’s components for metric aggregation, resource utilization analysis, and the generation of optimization proposals are equally applicable to other distributed systems as well. We want to abstract those core components in the long term and make them available to other projects as well. Our vision for Cruise Control is to build it in a manner that allows for straightforward integration with any distributed system to facilitate application-specific performance analysis, optimization, and execution.
Cruise Control began as an intern project by Efe Gencer. Many members on the Kafka development team have participated in brainstorming, design, and reviews. Cruise Control has also received many valuable contributions and insights from the Kafka SRE team at LinkedIn.
Editor’s Note: If you are interested in following some of the cool work we are doing in Stream Processing, both at LinkedIn and in our extended open source communities, you can register for our quarterly meetups where we invite speakers to talk about the latest improvements in Apache Kafka and Apache Samza.