How to Install ELK Stack on Ubuntu 20.04

In this article, we will have explained the necessary steps to install and configure ELK Stack on Ubuntu 20.04 LTS. Before continuing with this tutorial, make sure you are logged in as a user with sudo privileges. All the commands in this tutorial should be run as a non-root user.

ELK, currently known as Elastic Stack, is the acronym for open source projects comprising

• Elasticsearch is a search engine that provides a distributed, multitenant-capable full-text search engine and schema-free JSON documents across distributed sharded storage.
• Logstash is a free and open server-side data processing component that collects, parses, and transforms data before sending it to other sources, including Elasticsearch.
• Kibana is a free and open user interface that lets you explore and visualize Elasticsearch data. Beats are lightweight transport agents that collect application data and send it to Elasticsearch or another platform.

It is an open-source and one of the most popular log management platforms that collect processes and visualizes data from multiple data sources. It is mainly used for log analysis in IT environments. It is very helpful for a system administrator to search and analyze a large volume of data to make real-time decisions-all the time.

Install ELK Stack on Ubuntu 20.04

Step 1. First, before you start installing any package on your Ubuntu server, we always recommend making sure that all system packages are updated.

sudo apt update
sudo apt upgrade
sudo apt install wget apt-transport-https curl gnupg2

Step 2. Install Java.

Elasticsearch is a Java component and requires Java to be installed.  If you don’t have Java installed, install it by opening a terminal window and entering the following:

sudo apt install openjdk-11-jdk

Once all the packages are installed, verify the installed version of Java with the following command:

java -version

Step 3. Install Elasticsearch.

Run the following command to import the Elasticsearch public GPG key into APT:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Next, add the Elastic source list to the sources.list.d directory, where APT will look for new sources:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Then, install Elasticsearch with this command:

sudo apt update
sudo apt install elasticsearch

Now start and enable Elasticsearch services:

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Step 4. Install Kibana.

Run the commands below to install, start and enable Kibana services:

sudo apt install kibana
sudo systemctl start kibana
sudo systemctl enable kibana

Kibana provides a web interface that can be secured with a reverse proxy Kibana communicates over port 5601:

http://localhost:5601/status

Step 5. Install Logstash.

Logstash is the most popular log analysis platform and is responsible for aggregating data from different sources, processing it, and sending it down the pipeline, usually to be directly indexed in Elasticsearch. Run the commands below to install, start and enable its services:

sudo apt install logstash
sudo systemctl start logstash
sudo systemctl enable logstash

That should get Logstash installed and ready to be used. The default configuration of Logstash is found in /etc/logstash/conf.d:

sudo nano /etc/logstash/conf.d/02-beats-input.conf

Add the following file:

input {
  beats {
    port => 5044
  }
}

Then, create a file to define output to Elasticsearch:

sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf

Add the following file:

output {
  if [@metadata][pipeline] {
    elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    pipeline => "%{[@metadata][pipeline]}"
    }
  } else {
    elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

Make sure the configurations are ok, by running the validation commands below:

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

Step 6. Install Filebeat.

Filebeat is used to send logs to the Logstash or Elasticsearch for parsing. Run the commands below to install, start and enable Filebeat services:

sudo apt install filebeat
sudo systemctl start filebeat
sudo systemctl enable filebeat

Once the installation, follow the link below to configure Filebeat for data collection:

sudo nano /etc/filebeat/filebeat.yml

Add the following file:

# Configure what output to use when sending the data collected by the beat.# ---------------------------- Elasticsearch Output ---------------------------- #output.elasticsearch: # Array of hosts to connect to. # hosts: ["localhost:9200"] # Protocol - either `http` (default) or `https`. #protocol: "https" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" #username: "elastic" #password: "linuxtips890" # ------------------------------ Logstash Output ------------------------------- output.logstash: # The Logstash hosts hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications

Once done, run the commands below to enable Filebeat modules and parsing processes:

sudo filebeat modules enable system
sudo filebeat setup --pipelines --modules system

Load Filebeat template:

sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

Also, integrate Filebeat with Kibana:

sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601

Finally, Restart all components services:

sudo systemctl restart elasticsearch
sudo systemctl restart kibana
sudo systemctl restart logstash
sudo systemctl restart filebeat

Step 7. Access Kibana Web Interface.

You can access it using the URL http://your-server-ip:5601. You should see the Kibana dashboard on the following screen:

That’s all you need to do to install ELK on Ubuntu 20.04 LTS Focal Fossa. I hope you find this quick tip helpful. For further reading on ELK Stack, please refer to their official knowledge base. If you have questions or suggestions, feel free to leave a comment below.