Automated Kafka ops with Kafka Topology Builder and Gitops

01 Feb 2020

One of the common questions when teams start working with technologies like Apache Kafka is how to automate and simplify the common operations tasks. This task can be such as creating topics, managing their configuration or for example adding the necessary ACLs for a given application.

While many ways can get you to Rome, one of the options is to use Git, Jenkins and a bit of code to solve this problem. Gitops to the rescue. If you are not aware of this approach, I do recommend reading this very good explanation.

In a nutshell:

GitOps is a paradigm or a set of practices that empowers developers to perform tasks which typically fall under the purview of IT operations. GitOps requires us to describe and observe systems with declarative specifications that eventually form the basis of continuous everything.

Source https://www.cloudbees.com/gitops/what-is-gitops

So with this, we bring into the table:

• Autonomy for teams, they can request and fulfil their own ops needs independently.
• A declarative approach, users describe what they need and not how it is implemented, facilitating the solution of problems, keeping detailed implementation abstracted.
• A centralised way to verify and control changes, providing an audit trail to fulfil change management requirements.
• Have a centralised source of truth where the current state of the platform is represented

In this article we’re going to describe a possible solution for this situation, answering questions such as:

• How can we benefit from gitops with Apache Kafka and Confluent Platform?
• How to use it with Confluent Cloud?
• How to use it with your Kafka on-prem?

Kafka Ops with the Kafka Topology Builder

What is the Kafka Topology Builder

Kafka Topology builder is a tool where users can describe their needs (using a cluster description file(s)) and make sure their Apache Kafka cluster looks like it after applying the changes.

Based on a description file that can look like this:

A tool like this one can be a good starting point to build a GitOps strategy for Apache Kafka.

What can the Topology Builder do for you

With the Topology Builder a user can do, at the time of this article, topic and ACL management. In future releases we will include support for managing your Schemas within Schema Registry in an automated methodology.

Topic Management

One of the most common tasks when different teams require to use a Kafka Cluster is to manage their topic needs. Things like making sure the right namespacing policy, or a resilient configuration is in place, are complicated to control when teams grow without causing either failures or an unnecessary overload of work for operations.

Based on the structured provided to the tool, see as example: