In this article, we will have explained the necessary steps to install and setup Apache Kafka on Debian 10. Before continuing with this tutorial, make sure you are logged in as a user with sudo privileges. All the commands in this tutorial should be run as a non-root user.
Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for real-time handling data feeds. Kafka is highly valuable for enterprise infrastructures to process streaming data.
Install Apache Kafka on Debian
Step 1. The first command will update the package lists to ensure you get the latest version and dependencies.
sudo apt update sudo apt upgrade
Step 2. Install Java.
Apache Kafka is a Java application, so the first step is to install Java. Run following commands to install Java:
sudo apt install default-jdk
Verify the Java installation:
java -version
Step 3. Creating a User for Kafka.
To create a user, type the commands below while logged in as a non-root sudo user:
sudo useradd kafka -m sudo passwd kafka
Next, we need to add the kafka user to the sudo group for it to have sudo privileges that will be required to install Kafka’s dependencies:
sudo adduser kafka sudo su -l kafka
Step 4. Install Apache Kafka on Debian.
Download the Apache Kafka binary files from its official download website:
wget https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka-2.4.1-src.tgz
Then, extract the archive file:
tar xzf kafka-2.4.1-src.tgz mv kafka-2.4.1 /usr/local/kafka
Step 5. Configure the Kafka Server.
It is time to configure Apache Kafka. By default, we are not allowed to delete topics, categories or groups in which messages can be posted. To change this behavior, we need to edit the default configuration:
nano ~/kafka/config/server.properties
Add the following line to the bottom of the file to allow us to delete Kafka topics:
delete.topic.enable = true
Step 6. Setup Kafka Systemd Unit Files.
Create systemd unit files for the Zookeeper and Kafka service:
nano /etc/systemd/system/zookeeper.service
[Unit] Description=Apache Zookeeper server Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Now, let’s create a system unit file for kafka at the filepath /etc/systemd/system/kafka.service:
nano /etc/systemd/system/kafka.service
[Unit] Description=Apache Kafka Server Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64" ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target
Reload the systemd daemon to apply new changes:
systemctl daemon-reload
Next, start Kafka with the command below:
sudo systemctl enable kafka sudo systemctl start kafka
Step 7. Testing the Installation.
Kafka provides multiple pre-build shell script to work on it. First, create a topic named “testTopic” with a single partition with single replica:
$ cd /usr/local/kafka $ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic Created topic testTopic.
You can create multiple topics by running the same command as above. After that, you can see the created topics on Kafka by the running below command:
$ bin/kafka-topics.sh --list --zookeeper localhost:2181 testTopic LinuxtipsTut1 LinuxtipsTut2
Send Messages to Kafka:
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic >Welcome to kafka >This is my first topic >
Using Kafka Consumer:
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testTopic --from-beginning Welcome to kafka This is my first topic
Congratulation, you have learned how to install and configure Apache Kafka on Debian Buster. If you have any question, please leave a comment below.
