Graylog & ElasticSearch

Graylog is an open-source log management system that collects, indexes, and analyzes log data from various sources, providing real-time monitoring and alerting. Elasticsearch is a distributed search and analytics engine that efficiently indexes and queries large volumes of structured and unstructured data. Graylog often uses Elasticsearch as its storage and indexing backend for fast log searching and analysis. We will be installing the Graylog & Elastic Search on the Ubuntu 24.04LTS

---

Installing MongoDB Database (Community Edition)

Mongodb is the database system service which can be integrated by the graylog and which can be installed from the below given URL

https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-debian/

Installation Steps :

1. Importing the key pairs :

sudo apt-get install gnupg curl
curl -fsSL https://www.mongodb.org/static/pgp/server-8.0.asc | \
   sudo gpg -o /usr/share/keyrings/mongodb-server-8.0.gpg \
   --dearmor

1. Creating the list file

echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/8.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list

1. Reloading the packages database & installing mongodb

sudo apt-get update
sudo apt-get install -y mongodb-org

1. Registering services & enabling on startup

sudo systemctl start mongod
sudo systemctl enable mongod

Here we have installed the mongoDB database and now we will be installing elastic search.

---

Installing Elastic Search

1. Installing necessary packages

sudo apt install default-jdk apt-transport-https

1. Installing the elastic search GPG Key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

1. Add the Elasticsearch repository to your sources list:

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

1. Updating the packages & installing Elastic search

sudo apt update 
sudo apt install elasticsearch

Now the elastic search is installed and now we will configure the elastic search to enable the nodes & clusters to integrate with the graylog server

The configuration file of elasticsearch is located at /etc/elasticsearch/elasticsearch.yml we will add the following lines at the end of config file to enable the cluster :

sudo nano /etc/elasticsearch/elasticsearch.yml
## END OF THE CONFIG FILE
---------------------------------------
## CLUSTER CONFIG 
cluster.name: graylog
action.auto_create_index: false

We will also set the value of parameter xpack.security.enabled: false to disable security pack feature as it might not give outputs as expected.

sudo nano /etc/elasticsearch/elasticsearch.yml
# Enable security features
xpack.security.enabled: false

Now we will restart the service of elastic search

sudo systemctl restart elasticsearch

Now we will be creating the new password for the user elastic so that we can manage it using the default credentials.

cd /usr/share/elasticsearch && sudo ./bin/elasticsearch-reset-password -u elastic

This will ask the option to display the password as plain text in the console then press y and note the password displayed at the terminal. Now we will check wheather our elastic node is working perfect or not :

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:<PASSWORD_GENERATED> http://localhost:9200

The output should look like :

{
  "name" : "Legion",
  "cluster_name" : "graylog",
  "cluster_uuid" : "HY1TK3vQRb6_wCpKPNFHlg",
  "version" : {
    "number" : "8.17.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "747663ddda3421467150de0e4301e8d4bc636b0c",
    "build_date" : "2025-02-05T22:10:57.067596412Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Now we have successfully configured the elastic search node and now we will start configuring the graylog datanode & the graylog server.

---

Installing Graylog-datanode & Graylog Server

Firstly we will fetch the local repositories file from the graylog repo server

wget https://packages.graylog2.org/repo/packages/graylog-6.1-repository_latest.deb

Now we will install the .deb package which was downloaded above using

sudo dpkg -i <package_name_graylog>.deb

Once installed the debian package we will now update the repositories & install the necessary packages

sudo apt install -y graylog-server graylog-datanode

Once installed we will start configuring graylog server and integrate with the elastic search datanode which was deployed above.

To run Graylog Data Node, the value of the "vm.max_map_count" sysctl setting needs to be at least 262144 or more. Run the following commands to set and persist the value.

 sudo sysctl -w vm.max_map_count=262144

If the value is not the same we can do this by this command :

echo "vm.max_map_count = 262144" | sudo tee /etc/sysctl.d/99-graylog-datanode.conf

---

Configuring Graylog-server

The config file is located on /etc/graylog/server/server.conf , Firstly we will generate the 96 characters long random key and the encrypted password string to feed as input in the config file and then edit certian parameters to run the graylog server.

Generate a secret key for password hashing:

pwgen -N 1 -s 96

Generate an admin password hash:

echo -n 'YourPassword' | sha256sum

We will enter this vaules in the config files where the parameters are password_secret & root_password_sha2

## PASSWORD SECRET
password_secret = RPASN987AEGyMaatdfPRWQRXp5yTwhtDolY8UHxVA1wiNbzF5aaaaavruEBt0Bpz70356wMG30qRLO8SSAiUg8bza3t33vyr
## ROOT PASSWD
root_password_sha2 = 6f2cb9dd8f4b65e24e1c3f3fa5bc5aaaaa49237f11abceacd45bbcb74d621c25

Set Bind Address:

http_bind_address = 0.0.0.0:9000

Connect Graylog with Elasticsearch

elasticsearch_hosts = http://127.0.0.1:9200

Start & Enable services:

sudo systemctl daemon-reload
sudo systemctl enable --now graylog-server graylog-datanode
sudo systemctl restart graylog-server graylog-datanode

---

Configuring datanode

Create your password_secret with the following command:

< /dev/urandom tr -dc A-Z-a-z-0-9 | head -c${1:-96};echo;

Edit the config file:

sudo nano /etc/graylog/datanode/datanode.conf

Enter the key just generated in the password_secret to enable the datanode otherwise the datanode might not initialize

password_secret =DvM-BWV8HEMmKtxtzkG1VnWtVEL9PX9WaFfAICkNDdELcKTMsFYbs18oWpzBFYv42zGz

Ensure the following parameters are configured properly :

elasticsearch_hosts = http://127.0.0.1:9200

Save and restart:

sudo systemctl restart graylog-datanode

---

Accessing the Greylog UI

http://localhost:9000

---

Troubleshooting

Elastic search version 8.x not supported , Downgrade to 7.10.2

Step 1: Uninstall Elasticsearch 8.x

sudo systemctl stop elasticsearch
sudo apt remove --purge elasticsearch -y
sudo rm -rf /var/lib/elasticsearch /etc/elasticsearch

Step 2: Install Elasticsearch 7.10.2

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.2-amd64.deb
sudo dpkg -i elasticsearch-7.10.2-amd64.deb

Step 3: Configure Elasticsearch

Edit the config file:

sudo nano /etc/elasticsearch/elasticsearch.yml

Modify:

cluster.name: graylog
network.host: 127.0.0.1
http.port: 9200
discovery.type: single-node

Save and restart:

sudo systemctl enable --now elasticsearch

Step 4: Verify Elasticsearch

curl -X GET "http://127.0.0.1:9200"

Now restart Graylog:

sudo systemctl restart graylog-server

---