Install an ELK Server on Ubuntu 18.04

On By ProjectLeader

ELK Stack

The ELK stack is an amazing open source for logs and metrics.  What you can do with it is almost literally restricted by your imagination.  I use it extensively in my infrastructure for anything from ingesting server logs, setting alerts, statistic gathering and data visualization.  It is an amazing tool and worth the time to learn and add to your toolkit.

 

The ELK (or Elastic) stack has four main components which are:

  • Elasticsearch: a distributed RESTful search engine which stores all of the collected data.
  • Logstash: the data processing component of the Elastic Stack which sends incoming data to Elasticsearch.
  • Kibana: a web interface for searching and visualizing logs.
  • Beats: lightweight, single-purpose data shippers that can send data from hundreds or thousands of machines to either Logstash or Elasticsearch.

What you need:

  • Ubuntu 18.04 Server (or desktop) installed

First Step:    Install Java

$ sudo apt update
$ sudo apt install default-jre

Second Step:  Add Elasticsearch source and install

$ wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –
$ echo “deb https://artifacts.elastic.co/packages/6.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
$ sudo apt update
$ sudo apt install elasticsearch

Third Step:  Configure Elasticsearch to listen only on local host 

Change network.host value and save elasticsearch.yml

$ sudo nano /etc/elasticsearch/elasticsearch.yml

. . .
network.host: localhost
. . .

Start Elasticsearch:

$ sudo systemctl start elasticsearch

Start Elasticsearch on system boot:

$ sudo systemctl enable elasticsearch

Fourth Step:  Install Kibana

$ sudo apt install kibana

Enable and start Kibana

$ sudo systemctl enable kibana
$ sudo systemctl start kibana

Fifth Step:  Install Logstash

$ sudo apt install logstash

Configure an Beats input:

$ sudo nano /etc/logstash/conf.d/02-beats-input.conf

Add the following text

input {
beats {
port => 5044
}
}

Add a syslog input:

$ sudo nano /etc/logstash/conf.d/10-syslog-filter.conf

filter {
if [fileset][module] == “system” {
if [fileset][name] == “auth” {
grok {
match => { “message” => [“%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?”,
“%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}”,
“%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}”,
“%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: \s%{DATA:[system][auth][user]}  %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}”,                  “%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}”,                  “%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:[%{POSINT:[system][auth][pid]}])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$”,                  “%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[system][auth][pid]}])?: %{GREEDYMULTILINE:[system][auth][message]}”] }        pattern_definitions => {          “GREEDYMULTILINE”=> “(.|\n)”
}
remove_field => “message”
}
date {
match => [ “[system][auth][timestamp]”, “MMM  d HH:mm:ss”, “MMM dd HH:mm:ss” ]
}
geoip {
source => “[system][auth][ssh][ip]”
target => “[system][auth][ssh][geoip]”
}
}
else if [fileset][name] == “syslog” {
grok {
match => { “message” => [“%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}”] }
pattern_definitions => { “GREEDYMULTILINE” => “(.|\n)*” }
remove_field => “message”
}
date {
match => [ “[system][syslog][timestamp]”, “MMM  d HH:mm:ss”, “MMM dd HH:mm:ss” ]
}
}
}
}

Create Logstash Output

output {
elasticsearch {
hosts => [“localhost:9200”]
manage_template => false
index => “%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}”
}
}

Sixth Step:  Install Filebeat

$ sudo apt install filebeat

Configure Filebeat

$ sudo nano /etc/filebeat/filebeat.yml

Change the output.logstash section

output.logstash:
# The Logstash hosts
hosts: [“localhost:5044”]

Enable Filebeat

$ sudo filebeat modules enable system