Choosing the Right Kubernetes Operator for Apache Kafka

In the age of high-load, mission-critical applications, Apache Kafka has become an industry standard for queue management, event streaming, and real-time big data processing and analytics. Meanwhile, Kubernetes is a DevOps engineering favorite, attributing its position as the world’s leading cloud orchestration platform to a strong open-source foundation and powerful tools enabling automation, continuous delivery, and efficient container management.

Kubernetes was designed as a portable open-source system that helps automate the deployment, scaling, and management of containerized applications. It groups containers into logical units for easy management. This property of Kubernetes makes it advantageous for developers working with Apache Kafka.

In this article, we’ll pick from the extensive list of available Kubernetes operators to examine Koperator, Strimzi, and Confluent for Kafka. While these tools are relatively popular, developers should be aware of their strengths and weaknesses to make an informed decision.

Throughout this article, we’ll highlight notable features and areas where each operator excels or underperforms. But first, let’s discuss how operators help developers manage Apache Kafka and Kubernetes, and which factors to note when choosing operators.

What are Operators in Kubernetes?

We can significantly accelerate cluster growth by treating Apache Kafka clusters — called brokers — as applications within a single Kubernetes pod. This lets us add new brokers with Kubectl commands and lets us more easily change, update, and restart Kafka brokers.

To merge the two technologies, we need to extend the Kubernetes API with a series of controllers — called operators — that manage custom resources. These operators encode the domain- or application-specific functionality needed to automate an app’s entire lifecycle, including provisioning, scaling, and backup/restore functions.

Koperator

Koperator (formerly known as Banzai Cloud Kafka Operator) is an open-source core component of Banzai Cloud Supertubes. It was developed to provide solutions for highly dynamic environments, emphasizing graceful scaling, fine-grained broker configuration, and automatic self-adjustments in response to alerts from its plugin system and Prometheus.

Support and Development

We can use Koperator by itself for a lightweight, low-overhead solution. However, developers seeking more advanced features suited to a commercial production environment will need to supplement Koperator with more of the Banzai Cloud Supertubes suite.

Therefore, most of the compelling features and integrations below are only available as part of the Supertubes Core or Supertubes Pro product suites. Furthermore, many of the platform’s competitive managed components are featured only in the highest tier. Additionally, while support for the base Koperator component is limited to community resources, the community is active in maintaining the Banzai Cloud documentation.

For developers choosing a paid support package, Banzai Cloud offers several hours monthly of professional support. Banzai Cloud is now part of the Cisco conglomerate, so it is a fully capable enterprise solution.

Scalability

Banzai Cloud is a contributor to Kafka and has extensive experience operating Kafka at scale, which is reflected in its focus on configurable scalability. Koperator integrates LinkedIn’s Cruise-Control functionality to smoothly automate and manage deployments and workload rebalancing at large scales.

Supertubes supports both interactive and headless operation modes for ksqlDB, which it scales using a Horizontal Pod Autoscaler (HPA). To ensure that ksqlDB can accommodate high workloads, we can scale our HPAs while monitoring the consumer lag metric — which the integrated Prometheus instance already tracks — instead of relying on traditional CPU or memory usage metrics. Enabling this simply entails deploying a Helm chart and letting the preconfigured HPA do the work.

Setup and Ease of Use

Koperator provides basic functionality for automation when installing and managing Apache Kafka clusters. However, developers using the free version of Koperator will need to manually install Zookeeper, Prometheus, and any needed certification managers — either directly or using Helm. We can also deploy Koperator using a Helm chart.

Commercial versions of Koperator provide a smooth setup experience. After registering for a commercial version of Supertubes and installing the Supertubes CLI tool, you can install Koperator and its prerequisites with just the supertubes install -a command.

Installing components independently is more involved, but the Banzai Cloud documentation provides a full manual installation walkthrough.

Security

Koperator’s base security offering is similarly modest. For example, Koperator only offers basic functionality to manage Kafka ACLs. Developers looking for out-of-the-box mutual TLS (mTLS) will need to upgrade to the Supertubes Pro suite to take advantage of Istio operator integration.

Once upgraded, the Istio service mesh provides automatic mTLS authentication with built-in certificate rotation and management. This is faster than Kafka’s own built-in TLS implementation.

We can also apply Kafka ACLs and other Supertubes security features to our ksqlDB deployments by configuring authorization and access control in KsqlDB Custom Resource Definitions (CRDs). Accessing the ksqlDB from a CLI instance outside the Istio service mesh requires us to manually configure certificates.

We need to enable SSL encryption and generate our own security certificates in Apache Kafka clusters before they are created. For example, to create an Apache Kafka cluster with SSL encryption-enabled listeners and configure a certificate, you must configure your KafkaCluster Custom Resource.

Notably, once we create a cluster, we can’t change its listener configuration without incurring an outage.

Monitoring

Koperator optionally integrates Prometheus monitoring with the Grafana dashboard, collecting and collating data about important system metrics and displaying it in a graphical interface.

Strimzi

Strimzi is an open-source tool that helps manage and maintain Kafka clusters. Strimzi offers several operators, including ZooKeeper, Kafka Connect, Kafka MirrorMaker, and Kafka Exporter. The platform emphasizes deployment and management, with a focus on running Kafka components, managing brokers and users, and providing highly configurable access settings.

Support and Development

Strimzi is a Cloud Native Computing Foundation (CNCF) sandbox project. So, although its documentation is thorough and includes detailed guides with comprehensive background information, it lacks the commercial and enterprise options available to developers using Koperator. Support is likewise limited to what the community provides.

Another notable consideration for developers considering Strimzi is its history. The project spent a considerable amount of time struggling to move beyond its beta stage, so many developers are still wary of using it in production.

However, although the project remains heavily crowd-sourced with some documentation still in development, IBM and Red Hat have already adopted it, constituting a strong argument for its production readiness.

Setup and Ease of Use

Despite its incompleteness, the Strimzi documentation is one of this platform’s strongest elements. It features clear examples, use cases, and generously provisioned definitions and linked resources. These resources considerably flatten Strimzi’s learning curve, making it an attractive option that’s accessible to less experienced developers while retaining its utility in advanced use cases.

Strimzi presents an easy entry to its platform. Developers can consult the Quick Start guide for a brief refresher on Kafka and an overview of how Strimzi fits into their cloud architecture.

The installation for Strimzi is slightly more involved than that of its peers — even when compared to Koperator’s manual installation — but the procedure is, like all parts of Strimzi, clearly laid out. Developers will need to pay extra attention to how they manage namespaces to successfully deploy Strimzi.

Security

Strimzi offers stronger security features than Koperator’s free and intermediate tiers. Developers can secure listeners using mTLS authentication on TLS-enabled listeners or can implement SCRAM-SHA-512 or OAuth 2.0 token-based authentication to replace Strimzi’s default PLAIN mechanism.

If we choose OAuth 2.0 token-based authentication, Strimzi also lets us authorize Kafka brokers with OAuth 2.0. Otherwise, we can use simple or Open Policy Agent (OPA) authorization, or configure our own Authorizer plugin to define ACLs.

The operator offers its user management functionality through the command line. We can perform actions like viewing the list of users created in a cluster with a Kubectl command. Security configurations are stored in a secret Kubernetes resource, where the resource name is the same as the username.

The advantage of securing our Kafka deployments using this operator is its infrastructure-level approach. The platform lets us implement entry rules or a two-way authentication protocol, providing very strong security. However, this can result in significant transaction overhead.

Confluent

Confluent for Kubernetes (CFK) is a private cloud solution offering the benefits of a cloud-native experience on-premises. CFK handles data-in-motion workloads by automating and managing an opinionated deployment of Confluent Platform through a complete, declarative API.

If we can imagine that Koperator is a laptop for which we can purchase peripherals and more powerful internals, and that Strimzi is an engineering team’s shared, 3D-printed Raspberry Pi project, then CFK is the bespoke ergonomic home office of the IT director.

Support and Development

Created by the team that originally developed Apache Kafka, CFK adds enterprise-grade features to Kafka and automates common infrastructure lifecycle tasks.

CFK always includes the latest version of Kafka and provides a consistent experience for all major Kubernetes distributions, including services from Azure, Google, Amazon, Red Hat, VMWare, Rancher, and any Kubernetes managed service that conforms to CNCF standards.

Setup and Ease of Use

CFK equips developers with an infrastructure as code (IaC), Kubernetes-native declarative API toolkit to configure, deploy, and manage components and resources on Confluent Platform. This lets developers take advantage of ecosystem tools and features that are inherent to Kubernetes, rather than needing to build and maintain specialized deployment knowledge for things like Helm templates and storage configuration for stateful services.

The cornerstone of CFK’s approach is its emphasis on automation and resiliency. The platform performs automated updates for configuration changes, automated scaling with a single command, and automated rolling upgrades without introducing downtime.

Unlike Koperator and Strimzi, Confluent Control Center presents a comprehensive GUI-based way for us to create and manage clusters, set up topics, and inspect data streams without coding. By easing management and deployment this way, CFK makes it easier to develop Kubernetes applications and quickly derive actionable insights from the state of our deployments.

As developers, we also benefit from Confluent’s status as one of the world’s most recognized software providers. We can access a large user community for help and feedback, and we can connect a wide variety of data systems to CFK’s platform.

Security

CFK provides CRDs for developers to declaratively create and manage Confluent Role-Based Access Control (RBAC).

CFK also lets us perform post-deployment authentication management in Confluent Platform by updating existing Kubernetes secrets, either by having services pick up the updated user list without restarting the Kafka brokers or by performing a rolling update of Kafka.

By default, clients communicate to Kafka brokers on Confluent Platform by using the PLAINTEXT port, and the platform does not have any ACLs configured. Developers will need to implement and enable security features like encryption, network segmentation, authorization ACLs, or protocols like Kerberos to protect their deployment components

Conclusion

Developers can select from many different Kubernetes operators for Apache Kafka to handle high-load systems and improve automation and containerization of their processes. Koperator, Strimzi, and Confluent for Kafka present different approaches and toolsets to help make data processes scalable and reliable.

Koperator is a flexible Cisco-backed solution that’s available in a range of flavors, from a free lightweight core component to a fully managed suite at a premium.

Strimzi is a free community-backed solution offering an easy, comprehensively documented  onboarding process. It’s simple, highly configurable, and secure by default.

Confluent for Kubernetes is an opinionated solution that brings the cloud-native experience into the private cloud. It’s the priciest of the three operators, but provides smooth and convenient instrumentation backed by a distinguished pedigree.

Of course, we would like to see an operator solution that brings together the best of these offerings — something with container-granular storage, strong data security, insightful alerts backed by rigorous disaster recovery, and multi-cloud migrations.

Portworx provides these features alongside a full complement of sought-after operator features. This Kubernetes storage platform gives enterprises a robust foundation from which to confidently expand. Learn more about how to uncomplicate Data on Kubernetes with Portworx.