In this article, we will have explained the necessary steps to install and setup Apache Kafka on Debian 10. Before continuing with this tutorial, make sure you are logged in as a user with sudo privileges. All the commands in this tutorial should be run as a non-root user.
Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for real-time handling data feeds. Kafka is highly valuable for enterprise infrastructures to process streaming data.
Install Apache Kafka on Debian
Step 1. The first command will update the package lists to ensure you get the latest version and dependencies.
1 2 | sudoaptupdate sudoaptupgrade |
Step 2. Install Java.
Apache Kafka is a Java application, so the first step is to install Java. Run following commands to install Java:
1 | sudoaptinstalldefault-jdk |
Verify the Java installation:
1 | java-version |
Step 3. Creating a User for Kafka.
To create a user, type the commands below while logged in as a non-root sudo user:
1 2 | sudouseraddkafka-m sudopasswdkafka |
Next, we need to add the kafka user to the sudo group for it to have sudo privileges that will be required to install Kafka’s dependencies:
1 2 | sudoadduserkafkasudo su-lkafka |
Step 4. Install Apache Kafka on Debian.
Download the Apache Kafka binary files from its official download website:
1 | wgethttps://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka-2.4.1-src.tgz |
Then, extract the archive file:
1 2 | tarxzfkafka-2.4.1-src.tgz mvkafka-2.4.1/usr/local/kafka |
Step 5. Configure the Kafka Server.
It is time to configure Apache Kafka. By default, we are not allowed to delete topics, categories or groups in which messages can be posted. To change this behavior, we need to edit the default configuration:
1 | nano~/kafka/config/server.properties |
Add the following line to the bottom of the file to allow us to delete Kafka topics:
1 | delete.topic.enable=true |
Step 6. Setup Kafka Systemd Unit Files.
Create systemd unit files for the Zookeeper and Kafka service:
1 | nano/etc/systemd/system/zookeeper.service |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | [Unit] Description=ApacheZookeeperserver Documentation=http://zookeeper.apache.org Requires=network.targetremote-fs.target After=network.targetremote-fs.target [Service] Type=simple ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh/usr/local/kafka/config/zookeeper.properties ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target |
Now, let’s create a system unit file for kafka at the filepath /etc/systemd/system/kafka.service:
1 | nano/etc/systemd/system/kafka.service |
1 2 3 4 5 6 7 8 9 10 11 12 13 | [Unit] Description=ApacheKafkaServer Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64" ExecStart=/usr/local/kafka/bin/kafka-server-start.sh/usr/local/kafka/config/server.properties ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target |
Reload the systemd daemon to apply new changes:
1 | systemctldaemon-reload |
Next, start Kafka with the command below:
1 2 | sudosystemctlenablekafka sudosystemctlstartkafka |
Step 7. Testing the Installation.
Kafka provides multiple pre-build shell script to work on it. First, create a topic named “testTopic” with a single partition with single replica:
1 2 3 4 | $cd/usr/local/kafka $bin/kafka-topics.sh--create--zookeeperlocalhost:2181--replication-factor1--partitions1--topictestTopic CreatedtopictestTopic. |
You can create multiple topics by running the same command as above. After that, you can see the created topics on Kafka by the running below command:
1 2 3 4 5 | $bin/kafka-topics.sh--list--zookeeperlocalhost:2181 testTopic LinuxtipsTut1 LinuxtipsTut2 |
Send Messages to Kafka:
1 2 3 4 5 | $bin/kafka-console-producer.sh--broker-listlocalhost:9092--topictestTopic >Welcometokafka >Thisismyfirsttopic > |
Using Kafka Consumer:
1 2 3 4 | $bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictestTopic--from-beginning Welcometokafka Thisismyfirsttopic |
Congratulation, you have learned how to install and configure Apache Kafka on Debian Buster. If you have any question, please leave a comment below.
