Overview of Apache Kafka with Spring Boot

In simple terms, Apache Kafka is an open-source distributed publish-subscribe messaging platform that has been purpose-built to handle real-time streaming data. A kafka cluster consist of topics. Events are organized and durably stored in topics. In other words, a topic is similar to a folder in a filesystem, and the events are the files in that folder. Events in a topic are not deleted after consumption and can be retained according to a per-topic configuration setting. The older events are then discarded accordingly.

Overview of Kafka

Configuring Apache Kafka Server

Let us configure Apache Kafka Server now. We need to downloaded the Apache Kafka version 2.8.0 from https://kafka.apache.org/downloads. Download the zip file and extract its content. Rename the folder and copy it to the drive as shown below :

Kafka storage location in system

Now, we need to create kafka-logs and zookeeper-data folders and then edit the directory paths in zookeeper.properties and server.properties as shown below. This is done to store log files otherwise they will get stored in default tmp folder.

server.properties

zookeeper.properties

Also, you need to add listeners=PLAINTEXT://localhost:9092 otherwise the connection will not be established and a warning will be displayed.

WARN Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

server.properties

In this post, we are using .bat files to run the server on windows platform. You can use .sh files if you are running the server on linux/unix platform.

Spring Boot with Apache Kafka Producer

Now, we are going to learn how to use Spring Boot application to publish messages on Kafka topic. Open https://start.spring.io/ and add the configurations as mentioned in the image below. Here, we have added two dependencies Spring for Apache Kafka and Spring Web. Generate the jar to download the zip folder.

Configuration file in Spring Initializr

Extract the zip folder and import the maven project in Eclipse IDE or IntelliJ IDEA. We are using Eclipse IDE here. In the pom.xml file you can verify the dependencies.

pom.xml

In, application.properties set server.port=8080

Now, create a user resource to post the messages for kafka consumer. Here, we have created a user with Name, Department and ID. A simple string message can also be used.

UserResource.java

We need to start the Apache Kafka server before triggering our Spring Boot application. Apache Kafka contains zookeeper. It serves as the coordination interface between the Kafka broker and consumers. Zookeeper keeps track of status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions etc. In production, you might need to setup zookeeper separately. But here, we are using the zookeeper which comes with Apache Kafka.

Run C:\kafka>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Run C:\kafka>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties on command prompt. This will start the zookeeper service on port 2181.

Command Prompt to run Zookeeper

Open new command prompt terminal and run C:\kafka>.\bin\windows\kafka-server-start.bat .\config\server.properties to start Apache Kafka server.

Command Prompt to run Kafka server

Now, open another terminal to receive messages in kafka consumer console.

The below command creates a topic called Kafka_Example with single topic and only one replica :

C:\kafka>.\bin\windows\kafka-topics.bat — create — zookeeper localhost:2181 — replication-factor 1 — partitions 1 — topic Kafka_Example

Command Prompt to create Kafka topic and run Kafka Consumer via console

Kafka also provides a command-line consumer client for message consumption. The following command is used to start the console-based consumer :

C:\kafka>.\bin\windows\kafka-console-consumer.bat — bootstrap-server localhost:9092 — topic Kafka_Example — from-beginning

Now publish the user name on localhost then you can observe that Kafka consumer console displaying the details of those user.

http://localhost:8080/kafka/publish/Maria

http://localhost:8080/kafka/publish/John

Kafka Consumer displays the details of those user

Spring Boot with Apache Kafka Consumer

Now, we will see how to use Spring Boot application to consume messages from a Kafka topic. We need to create another project for consumer. The same way we created for producer.

Configuration file in Spring Initializr

After creating and setting up the project in Eclipse. Here, I have created a sample Kafka configuration file which sends messages in String.

KafkaConfiguration.java

KafkaConsumer is used to read data from Kafka server.

KafkaConsumer.java

Now, kick start your spring boot application and then start zookeeper and Kafka server in separate command prompt. As we have already created a Kafka topic Kafka_Example. We need not create this topic again. But if you are using a new Kafka topic then a new topic has to be created.

Kafka provides users with a command-line producer client that accepts inputs from the command line and publishes them as a message to the Kafka cluster. By default each new line entered is considered as a new message. The following command is used for starting the console-based producer for sending messages :

Kafka Producer sends messages

Now, after running Kafka console producer in separate command prompt, the messages can be published. These messages are then consumed in our spring boot application.

Spring Boot application displays the messages which are published by Kafka Producer

This is how Spring Boot application reads data from Apache Kafka.