How to Install Apache Kafka on Debian 10

Install Apache Kafka on Debian 10

In this article, we will have explained the necessary steps to install and setup Apache Kafka on Debian 10. Before continuing with this tutorial, make sure you are logged in as a user with sudo privileges. All the commands in this tutorial should be run as a non-root user.

Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for real-time handling data feeds. Kafka is highly valuable for enterprise infrastructures to process streaming data.

Install Apache Kafka on Debian

Step 1. The first command will update the package lists to ensure you get the latest version and dependencies.

sudo apt update
sudo apt upgrade

Step 2. Install Java.

Apache Kafka is a Java application, so the first step is to install Java. Run following commands to install Java:

sudo apt install default-jdk

Verify the Java installation:

java -version

Step 3. Creating a User for Kafka.

To create a user, type the commands below while logged in as a non-root sudo user:

sudo useradd kafka -m
sudo passwd kafka

Next, we need to add the kafka user to the sudo group for it to have sudo privileges that will be required to install Kafka’s dependencies:

sudo adduser kafka sudo
su -l kafka

Step 4. Install Apache Kafka on Debian.

Download the Apache Kafka binary files from its official download website:

wget https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka-2.4.1-src.tgz

Then, extract the archive file:

tar xzf kafka-2.4.1-src.tgz
mv kafka-2.4.1 /usr/local/kafka

Step 5. Configure the Kafka Server.

It is time to configure Apache Kafka. By default, we are not allowed to delete topics, categories or groups in which messages can be posted. To change this behavior, we need to edit the default configuration:

nano ~/kafka/config/server.properties

Add the following line to the bottom of the file to allow us to delete Kafka topics:

delete.topic.enable = true

Step 6. Setup Kafka Systemd Unit Files.

Create systemd unit files for the Zookeeper and Kafka service:

nano /etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Now, let’s create a system unit file for kafka at the filepath /etc/systemd/system/kafka.service:

nano /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Reload the systemd daemon to apply new changes:

systemctl daemon-reload

Next, start Kafka with the command below:

sudo systemctl enable kafka
sudo systemctl start kafka

Step 7. Testing the Installation.

Kafka provides multiple pre-build shell script to work on it. First, create a topic named “testTopic” with a single partition with single replica:

$ cd /usr/local/kafka
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic

Created topic testTopic.

You can create multiple topics by running the same command as above. After that, you can see the created topics on Kafka by the running below command:

$ bin/kafka-topics.sh --list --zookeeper localhost:2181

testTopic
LinuxtipsTut1
LinuxtipsTut2

Send Messages to Kafka:

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic

>Welcome to kafka
>This is my first topic
>

Using Kafka Consumer:

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testTopic --from-beginning

Welcome to kafka
This is my first topic

Congratulation, you have learned how to install and configure Apache Kafka on Debian Buster. If you have any question, please leave a comment below.