Choosing the Right Kubernetes Operator for Apache Kafka

In the age of high-load, mission-critical applications, Apache Kafka has become an industry standard for queue management, event streaming, and real-time big data processing and analytics. Meanwhile, Kubernetes is a DevOps engineering favorite, attributing its position as the world’s leading cloud orchestration platform to a strong open-source foundation and powerful tools enabling automation, continuous delivery, and efficient container management.

Kubernetes was designed as a portable open-source system that helps automate the deployment, scaling, and management of containerized applications. It groups containers into logical units for easy management. This property of Kubernetes makes it advantageous for developers working with Apache Kafka.

In this article, we’ll pick from the extensive list of available Kubernetes operators to examine Strimzi, and Confluent for Kafka. While these tools are relatively popular, developers should be aware of their strengths and weaknesses to make an informed decision.

Throughout this article, we’ll highlight notable features and areas where each operator excels or underperforms. But first, let’s discuss how operators help developers manage Apache Kafka and Kubernetes, and which factors to note when choosing operators.

What are Operators in Kubernetes?

We can significantly accelerate cluster growth by treating Apache Kafka clusters — called brokers — as applications within a single Kubernetes pod. This lets us add new brokers with Kubectl commands and lets us more easily change, update, and restart Kafka brokers.

To merge the two technologies, we need to extend the Kubernetes API with a series of controllers — called operators — that manage custom resources. These operators encode the domain- or application-specific functionality needed to automate an app’s entire lifecycle, including provisioning, scaling, and backup/restore functions.

Strimzi

Strimzi is an open-source tool that helps manage and maintain Kafka clusters. Strimzi offers several operators, including ZooKeeper, Kafka Connect, Kafka MirrorMaker, and Kafka Exporter. The platform emphasizes deployment and management, with a focus on running Kafka components, managing brokers and users, and providing highly configurable access settings.

Support and Development

Strimzi is a Cloud Native Computing Foundation (CNCF) sandbox project. So, although its documentation is thorough and includes detailed guides with comprehensive background information, it lacks commercial and enterprise options. Support is likewise limited to what the community provides.

Another notable consideration for developers considering Strimzi is its history. The project spent a considerable amount of time struggling to move beyond its beta stage, so many developers are still wary of using it in production.

However, although the project remains heavily crowd-sourced with some documentation still in development, IBM and Red Hat have already adopted it, constituting a strong argument for its production readiness.

Setup and Ease of Use

Despite its incompleteness, the Strimzi documentation is one of this platform’s strongest elements. It features clear examples, use cases, and generously provisioned definitions and linked resources. These resources considerably flatten Strimzi’s learning curve, making it an attractive option that’s accessible to less experienced developers while retaining its utility in advanced use cases.

Strimzi presents an easy entry to its platform. Developers can consult the Quick Start guide for a brief refresher on Kafka and an overview of how Strimzi fits into their cloud architecture.

The installation for Strimzi is slightly more involved than that of its peers, but the procedure is, like all parts of Strimzi, clearly laid out. Developers will need to pay extra attention to how they manage namespaces to successfully deploy Strimzi.

Security

Developers can secure listeners using mTLS authentication on TLS-enabled listeners or can implement SCRAM-SHA-512 or OAuth 2.0 token-based authentication to replace Strimzi’s default PLAIN mechanism.

If we choose OAuth 2.0 token-based authentication, Strimzi also lets us authorize Kafka brokers with OAuth 2.0. Otherwise, we can use simple or Open Policy Agent (OPA) authorization, or configure our own Authorizer plugin to define ACLs.

The operator offers its user management functionality through the command line. We can perform actions like viewing the list of users created in a cluster with a Kubectl command. Security configurations are stored in a secret Kubernetes resource, where the resource name is the same as the username.

The advantage of securing our Kafka deployments using this operator is its infrastructure-level approach. The platform lets us implement entry rules or a two-way authentication protocol, providing very strong security. However, this can result in significant transaction overhead.

Confluent

Confluent for Kubernetes (CFK) is a private cloud solution offering the benefits of a cloud-native experience on-premises. CFK handles data-in-motion workloads by automating and managing an opinionated deployment of Confluent Platform through a complete, declarative API.

Support and Development

Created by the team that originally developed Apache Kafka, CFK adds enterprise-grade features to Kafka and automates common infrastructure lifecycle tasks.

CFK always includes the latest version of Kafka and provides a consistent experience for all major Kubernetes distributions, including services from Azure, Google, Amazon, Red Hat, VMWare, Rancher, and any Kubernetes managed service that conforms to CNCF standards.

Setup and Ease of Use

CFK equips developers with an infrastructure as code (IaC), Kubernetes-native declarative API toolkit to configure, deploy, and manage components and resources on Confluent Platform. This lets developers take advantage of ecosystem tools and features that are inherent to Kubernetes, rather than needing to build and maintain specialized deployment knowledge for things like Helm templates and storage configuration for stateful services.

The cornerstone of CFK’s approach is its emphasis on automation and resiliency. The platform performs automated updates for configuration changes, automated scaling with a single command, and automated rolling upgrades without introducing downtime.

Unlike Strimzi, Confluent Control Center presents a comprehensive GUI-based way for us to create and manage clusters, set up topics, and inspect data streams without coding. By easing management and deployment this way, CFK makes it easier to develop Kubernetes applications and quickly derive actionable insights from the state of our deployments.

As developers, we also benefit from Confluent’s status as one of the world’s most recognized software providers. We can access a large user community for help and feedback, and we can connect a wide variety of data systems to CFK’s platform.

Security

CFK provides CRDs for developers to declaratively create and manage Confluent Role-Based Access Control (RBAC).

CFK also lets us perform post-deployment authentication management in Confluent Platform by updating existing Kubernetes secrets, either by having services pick up the updated user list without restarting the Kafka brokers or by performing a rolling update of Kafka.

By default, clients communicate to Kafka brokers on Confluent Platform by using the PLAINTEXT port, and the platform does not have any ACLs configured. Developers will need to implement and enable security features like encryption, network segmentation, authorization ACLs, or protocols like Kerberos to protect their deployment components

Conclusion

Developers can select from many different Kubernetes operators for Apache Kafka to handle high-load systems and improve automation and containerization of their processes. Strimzi, and Confluent for Kafka present different approaches and toolsets to help make data processes scalable and reliable.

Strimzi is a free community-backed solution offering an easy, comprehensively documented onboarding process. It’s simple, highly configurable, and secure by default.

Confluent for Kubernetes is an opinionated solution that brings the cloud-native experience into the private cloud. It’s the priciest of the three operators, but provides smooth and convenient instrumentation backed by a distinguished pedigree.

Of course, we would like to see an operator solution that brings together the best of these offerings — something with container-granular storage, strong data security, insightful alerts backed by rigorous disaster recovery, and multi-cloud migrations.

Portworx provides these features alongside a full complement of sought-after operator features. This Kubernetes storage platform gives enterprises a robust foundation from which to confidently expand. Learn more about how to uncomplicate Data on Kubernetes with Portworx.