How to start Spring Kafka Application with Spring Boot

The Spring Apache Kafka (spring-kafka) provides a high-level abstraction for Kafka-based messaging solutions. And in the previous post, We had developed a Spring Kafka Application with the auto-configuration supported by SpringBoot (from version 1.5). But when we need explicitly configure Kafka factories (Kafka Producer and Kafka Consumer) for development, how to do it? So in the tutorial, JavaSampleApproach will introduce an alternative solution by manually configure Kafka factories to build a Spring Kafka Application.

Related Articles:
How to start Apache Kafka
How to start Spring Apache Kafka Application with SpringBoot Auto-Configuration
How to use Spring Kafka JsonSerializer(JsonDeserializer) to produce/consume Java Object messages
How to create Spring RabbitMQ Producer/Consumer application with SpringBoot


I. Technologies

– Java 8
– Maven build
– Spring Boot
– Spring Kafka
– Apache Kafka
– Spring Tool Suite editor

II. Overview

Spring Kafka Application - architecture

We will explicitly implement a ProducerFactory and ConsumerFactory with customized properties:

Then use ProducerFactory to build KafkaTemplate and use ConsumerFactory to build
ConcurrentKafkaListenerContainerFactory which will handle @KafkaListener later:

III. Practice

We create a SpringBoot project with 2 main services: KafkaProducer and KafkaConsumer for sending and receiving messages from Apache Kafka cluster.
And export 2 RestAPIs {‘/producer’, ‘/consumer’} for interaction.

Spring Kafka Application - project structure

Step to do:
– Create a SpringBoot project
– Create Kafa Factories (ProducerFactory & ConsumerFactory)
– Create Services (Producer and Consumer)
– Export some RestAPIs
– Deployment

1. Create a SpringBoot project

Use SpringToolSuite to create a SpringBoot project, then add dependencies {spring-kafka, spring-boot-starter-web}:

2. Create Kafa Factories (ProducerFactory & ConsumerFactory)

Open application.properties, add kafka configuration:

jsa.kafka.bootstrap-servers is used to indicate the Kafka Cluster address.
jsa.kafka.consumer.group-id is used to indicate the consumer-group-id.
jsa.kafka.topic is used to define a Kafka topic name to produce and receive messages.

2.1 Create ProducerFactory and KafkaTemplate

2.2 Create ConsumerFactory and KafkaListenerContainerFactory

@EnableKafka is used to enable detection of @KafkaListener annotation.

3. Create Services (Producer and Consumer)

– Create a KafkaProducer service:

– Create a KafkaConsumer service:

About MessageStorage, it is an additional implement to store Kafka-based messages after received. See details the implementation of MessageStorage:

4. Export some RestAPIs

Create a Web Controller to export 2 RestAPIs {‘/producer’, ‘/consumer’}

/producer is used to send messages from browser to KafkaProducer service.
/consumer is used to get all recieved messages that are buffered in MessageStorage.

5. Deployment

Start Apache Kafka Cluster:
– Start a ZooKeeper:

– Start the Apache Kafka server:

>>> More details at: How to start Apache Kafka

Build and Install the SpringBoot project with commandlines: mvn clean install and mvn spring-boot:run

– Make a producer request: http://localhost:8080/jsa/kafka/producer?data=Hello World
-> Logs:

– Make another producer request: http://localhost:8080/jsa/kafka/producer?data=This is a SpringBoot Kafka Application

-> Logs:

– Make a consumer request: http://localhost:8080/jsa/kafka/consumer, result:

Spring Kafka Application - results

IV. Sourcecode

SpringApacheKafka

By JavaSampleApproach | June 10, 2017.

Related Posts


9 thoughts on “How to start Spring Kafka Application with Spring Boot”

  1. Thanks , well explained. But i was facing an error while sending message. Its timed out after long waiting to send message. Any thoughts?

    My propeties here:

    1. Hi Venkat,

      I had tried to re-produce your case. But can NOT in my environment.
      Do you try the tutorial with your local enviroment or any cloud or docker?

      The time-out exception may be related with your server Kafka setup. I had the experience when done with other tech-stacks.
      May be, your cursor in cmd stops the excution of Kafka engine -> So please check it by do an enter keyboard on Kafka starting server cmd.

      If having any difficult for setup and running, you can follow the video guide: https://youtu.be/A2FXupo7FLs

      Regards,
      JSA

  2. I am getting the following error while doing mvn spring-boot:run.Any help??

    java.net.BindException: Address already in use: bind
    at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_111]
    at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_111]
    at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_111]
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[na:1.8.0_111]
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.8.0_111]
    at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:210) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:978) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:628) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:993) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.StandardService.addConnector(StandardService.java:225) [tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.addPreviouslyRemovedConnectors(TomcatEmbeddedServletContainer.java:247) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.start(TomcatEmbeddedServletContainer.java:190) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.startEmbeddedServletContainer(EmbeddedWebApplicationContext.java:297) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:145) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545) [spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1162) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1151) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at com.javasampleapproach.apachekafka.SpringBootApacheKafkaApplication.main(SpringBootApacheKafkaApplication.java:10) [classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
    at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:527) [spring-boot-maven-plugin-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

    2018-02-16 14:45:25.381 ERROR 9564 — [ main] o.apache.catalina.core.StandardService : Failed to start connector [Connector[HTTP/1.1-8080]]

    org.apache.catalina.LifecycleException: Failed to start component [Connector[HTTP/1.1-8080]]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.core.StandardService.addConnector(StandardService.java:225) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.addPreviouslyRemovedConnectors(TomcatEmbeddedServletContainer.java:247) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.start(TomcatEmbeddedServletContainer.java:190) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.startEmbeddedServletContainer(EmbeddedWebApplicationContext.java:297) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:145) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545) [spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1162) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1151) [spring-boot-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at com.javasampleapproach.apachekafka.SpringBootApacheKafkaApplication.main(SpringBootApacheKafkaApplication.java:10) [classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
    at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:527) [spring-boot-maven-plugin-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
    Caused by: org.apache.catalina.LifecycleException: service.getName(): “Tomcat”; Protocol handler start failed
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:1000) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    … 19 common frames omitted
    Caused by: java.net.BindException: Address already in use: bind
    at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_111]
    at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_111]
    at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_111]
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[na:1.8.0_111]
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.8.0_111]
    at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:210) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:978) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:628) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:993) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
    … 20 common frames omitted

    2018-02-16 14:45:25.431 INFO 9564 — [ main] o.apache.catalina.core.StandardService : Stopping service Tomcat
    2018-02-16 14:45:25.449 INFO 9564 — [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : Discovered coordinator D-113060917.wipro.com:9092 (id: 2147483647 rack: null) for group jsa-group.
    2018-02-16 14:45:25.462 INFO 9564 — [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : Revoking previously assigned partitions [] for group jsa-group
    2018-02-16 14:45:25.464 INFO 9564 — [ main] utoConfigurationReportLoggingInitializer :

    Error starting ApplicationContext. To display the auto-configuration report re-run your application with ‘debug’ enabled.
    2018-02-16 14:45:25.469 INFO 9564 — [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions revoked:[]
    2018-02-16 14:45:25.472 INFO 9564 — [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : (Re-)joining group jsa-group
    2018-02-16 14:45:25.476 ERROR 9564 — [ main] o.s.b.d.LoggingFailureAnalysisReporter :

    ***************************
    APPLICATION FAILED TO START
    ***************************

    Description:

    The Tomcat connector configured to listen on port 8080 failed to start. The port may already be in use or the connector may be misconfigured.

    Action:

    Verify the connector’s configuration, identify and stop any process that’s listening on port 8080, or configure this application to listen on another port.

    2018-02-16 14:45:25.498 INFO 9564 — [ main] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@4f055159: startup date [Fri Feb 16 14:45:22 IST 2018]; root of context hierarchy
    2018-02-16 14:45:25.503 INFO 9564 — [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 0
    2018-02-16 14:45:25.508 INFO 9564 — [ntainer#0-0-C-1] essageListenerContainer$ListenerConsumer : Consumer stopped
    2018-02-16 14:45:25.508 INFO 9564 — [ main] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
    [WARNING]
    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:527)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: org.springframework.boot.context.embedded.tomcat.ConnectorStartFailedException: Connector configured to listen on port 8080 failed to start
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.checkThatConnectorsHaveStarted(TomcatEmbeddedServletContainer.java:219)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.start(TomcatEmbeddedServletContainer.java:195)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.startEmbeddedServletContainer(EmbeddedWebApplicationContext.java:297)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:145)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1162)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1151)
    at com.javasampleapproach.apachekafka.SpringBootApacheKafkaApplication.main(SpringBootApacheKafkaApplication.java:10)
    … 6 more
    [INFO] ————————————————————————
    [INFO] BUILD FAILURE
    [INFO] ————————————————————————
    [INFO] Total time: 8.289 s
    [INFO] Finished at: 2018-02-16T14:45:25+05:30
    [INFO] Final Memory: 36M/274M
    [INFO] ————————————————————————
    [ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:1.5.3.RELEASE:run (default-cli) on project SpringBootApacheKafka: An exception occurred while running. null: InvocationTargetException: Connector configured to listen on port 8080 failed to start -> [Help 1]
    [ERROR]
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

    1. The error messages are clearly telling you that you have a port conflict as something else on your machine is using the same port (8080). Stop that process and try again.

Got Something To Say:

Your email address will not be published. Required fields are marked *

*