How to start Spring Kafka Application with Spring Boot

The Spring Apache Kafka (spring-kafka) provides a high-level abstraction for Kafka-based messaging solutions. And in the previous post, We had developed a Spring Kafka Application with the auto-configuration supported by SpringBoot (from version 1.5). But when we need explicitly configure Kafka factories (Kafka Producer and Kafka Consumer) for development, how to do it? So in the tutorial, JavaSampleApproach will introduce an alternative solution by manually configure Kafka factories to build a Spring Kafka Application.

Related Articles:
How to start Apache Kafka
How to start Spring Apache Kafka Application with SpringBoot Auto-Configuration
How to use Spring Kafka JsonSerializer(JsonDeserializer) to produce/consume Java Object messages
How to create Spring RabbitMQ Producer/Consumer application with SpringBoot


I. Technologies

– Java 8
– Maven build
– Spring Boot
– Spring Kafka
– Apache Kafka
– Spring Tool Suite editor

II. Overview

Spring Kafka Application - architecture

We will explicitly implement a ProducerFactory and ConsumerFactory with customized properties:

Then use ProducerFactory to build KafkaTemplate and use ConsumerFactory to build
ConcurrentKafkaListenerContainerFactory which will handle @KafkaListener later:

III. Practice

We create a SpringBoot project with 2 main services: KafkaProducer and KafkaConsumer for sending and receiving messages from Apache Kafka cluster.
And export 2 RestAPIs {‘/producer’, ‘/consumer’} for interaction.

Spring Kafka Application - project structure

Step to do:
– Create a SpringBoot project
– Create Kafa Factories (ProducerFactory & ConsumerFactory)
– Create Services (Producer and Consumer)
– Export some RestAPIs
– Deployment

1. Create a SpringBoot project

Use SpringToolSuite to create a SpringBoot project, then add dependencies {spring-kafka, spring-boot-starter-web}:

2. Create Kafa Factories (ProducerFactory & ConsumerFactory)

Open application.properties, add kafka configuration:

jsa.kafka.bootstrap-servers is used to indicate the Kafka Cluster address.
jsa.kafka.consumer.group-id is used to indicate the consumer-group-id.
jsa.kafka.topic is used to define a Kafka topic name to produce and receive messages.

2.1 Create ProducerFactory and KafkaTemplate

2.2 Create ConsumerFactory and KafkaListenerContainerFactory

@EnableKafka is used to enable detection of @KafkaListener annotation.

3. Create Services (Producer and Consumer)

– Create a KafkaProducer service:

– Create a KafkaConsumer service:

About MessageStorage, it is an additional implement to store Kafka-based messages after received. See details the implementation of MessageStorage:

4. Export some RestAPIs

Create a Web Controller to export 2 RestAPIs {‘/producer’, ‘/consumer’}

/producer is used to send messages from browser to KafkaProducer service.
/consumer is used to get all recieved messages that are buffered in MessageStorage.

5. Deployment

Start Apache Kafka Cluster:
– Start a ZooKeeper:

– Start the Apache Kafka server:

>>> More details at: How to start Apache Kafka

Build and Install the SpringBoot project with commandlines: mvn clean install and mvn spring-boot:run

– Make a producer request: http://localhost:8080/jsa/kafka/producer?data=Hello World
-> Logs:

– Make another producer request: http://localhost:8080/jsa/kafka/producer?data=This is a SpringBoot Kafka Application

-> Logs:

– Make a consumer request: http://localhost:8080/jsa/kafka/consumer, result:

Spring Kafka Application - results

IV. Sourcecode

SpringApacheKafka


Related Posts


2 thoughts on “How to start Spring Kafka Application with Spring Boot”

  1. Thanks , well explained. But i was facing an error while sending message. Its timed out after long waiting to send message. Any thoughts?

    My propeties here:

    1. Hi Venkat,

      I had tried to re-produce your case. But can NOT in my environment.
      Do you try the tutorial with your local enviroment or any cloud or docker?

      The time-out exception may be related with your server Kafka setup. I had the experience when done with other tech-stacks.
      May be, your cursor in cmd stops the excution of Kafka engine -> So please check it by do an enter keyboard on Kafka starting server cmd.

      If having any difficult for setup and running, you can follow the video guide: https://youtu.be/A2FXupo7FLs

      Regards,
      JSA

Got Something To Say:

Your email address will not be published. Required fields are marked *

*