Managing multiple Wazuh clusters with Cross-Cluster Search

PreviousSingle node deployment with Script NextCluster Configuration

Last updated 2 months ago

Managing multiple Wazuh clusters with Cross-Cluster Search

Cross-Cluster Search (CCS) in Wazuh enables querying and viewing alerts from remote Wazuh clusters at a centralized location, while keeping log data at the source cluster. This setup is trusted by remote clusters and allows security alerts to be monitored via a single Wazuh dashboard, even in a remote Security Operations Center (SOC).

Use Case:

A cybersecurity operations company provides managed SOC services to two customers (Organizations A and B). Each customer requires separate log management with tightly controlled access and data retention within their own environment.

Solution Overview:

CCS Environment: This environment is used only for querying data from remote Wazuh clusters. It contains the Wazuh indexer and dashboard but no agents or Wazuh server. It is trusted by clusters A and B.
Cluster A: Contains a Wazuh server and indexer for Organization A, with no dashboard (visualization occurs in the CCS environment). It trusts the CCS environment for data access.
Cluster B: Contains a Wazuh server and indexer for Organization B, also with no dashboard, but trusts the CCS environment for data access.

The table below shows a further breakdown of the infrastructure.

Cluster

Component

Node name

IP address

CCS

Wazuh dashboard

ccs-wazuh-dashboard

192.168.146.154

Wazuh indexer

ccs-wazuh-indexer-1

Cluster A

Wazuh server

ca-wazuh-server-1

192.168.146.155

Wazuh indexer

ca-wazuh-indexer-1

Cluster B

Wazuh server

cb-wazuh-server-1

192.168.146.156

Wazuh indexer

cb-wazuh-indexer-1

Ensure network connectivity between the remote clusters and the CCS environment, irrespective of the network address.

CSS Environment Configuration :

A Cross-Cluster Search (CCS) environment is authorized to perform search operations on remote Wazuh clusters. It is typically made up of the Wazuh indexer and the Wazuh dashboard. The Wazuh indexer node establishes trust with indexer nodes in remote clusters, enabling it to search for log data on these indexers. The data on the Wazuh indexer nodes in the remote clusters are displayed on the Wazuh dashboard in the CCS environment.

Generating certificates

Download the wazuh-certs-tool.sh script and the config.yml configuration file. The wazuh-certs-tool.sh script is used to generate certificates for the cluster. The config.yml file defines the IP addresses and node names of the Wazuh central components to be deployed.

# ENSURE YOU ARE CHANGING URLS BASED ON LATEST STABLE VERSION OF WAZUH
curl -sO https://packages.wazuh.com/4.9/config.yml
curl -sO https://packages.wazuh.com/4.9/wazuh-certs-tool.sh

Edit the config.yml file and replace the name and IP variables with the corresponding node names and IP addresses:

nodes:
  # Wazuh indexer nodes
  indexer:
    - name: ccs-wazuh-indexer-1
      ip: "192.168.146.154"
  # Wazuh dashboard nodes
  dashboard:
    - name: ccs-wazuh-dashboard
      ip: "192.168.146.154"

Note: Comment out or remove the Wazuh server configuration since it is not in use.

Run the wazuh-certs-tool.sh script with option -A to create the root-ca, admin, and node certificates:

bash ./wazuh-certs-tool.sh -A
# THE OUTPUT SHOULD LOOK LIKE 
25/08/2024 15:22:57 INFO: Generating the root certificate.
25/08/2024 15:22:57 INFO: Generating Admin certificates.
25/08/2024 15:22:58 INFO: Admin certificates created.
25/08/2024 15:22:58 INFO: Generating Wazuh indexer certificates.
25/08/2024 15:22:58 INFO: Wazuh indexer certificates created.
25/08/2024 15:22:58 INFO: Generating Wazuh dashboard certificates.
25/08/2024 15:22:59 INFO: Wazuh dashboard certificates created.

Copy the root CA certificate to the working directory of a node in cluster A and cluster B using the scp utility. The root CA is used later in the remote clusters to generate certificates.

Here we will pass the root certificates file into the nodes to ensure the connections are encrypted between master & worker nodes

scp -r wazuh-certificates/root-ca.* <USER_NAME>@<IP_ADDRESS>:<DESTINATION_DIRECTORY>
## MY COMMANDS
#scp -r wazuh-certificates/root-ca.* clustera@192.168.146.155:/home/clustera
#scp -r wazuh-certificates/root-ca.* clustern@192.168.146.156:/home/clusterb

Compress all the certificate files and remove the uncompressed version to allow for easier transfer to other component nodes within the CCS environment if need be:

tar -cvf ./wazuh-certificates.tar -C ./wazuh-certificates/ .

Installing & configuring Wazuh indexer

Perform the following configuration steps on the Wazuh indexer node for the CCS environment. In my case the machine I used is ubuntu , you can install the indexer from the manual for any other distributions of linux

apt-get install debconf adduser procps
## ADDING WAZUH REPO
apt-get install gnupg apt-transport-httpscurl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import && chmod 644 /usr/share/keyrings/wazuh.gpg
## ADDING REPO SOURCE FILE
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
## UPDATING PACKAGES 
apt-get update

Installing Wazuh Indexer

apt-get -y install wazuh-indexer

Edit the /etc/wazuh-indexer/opensearch.yml configuration file:

network.host: "192.168.146.154" <-- CHANGE THIS -->
node.name: "ccs-wazuh-indexer-1" <-- CHANGE THIS ->

cluster.initial_master_nodes:
- "ccs-wazuh-indexer-1" <-- CHANGE THIS --> 
cluster.name: "ccs-cluster" <-- CHANGE THIS --> 
node.max_local_storage_nodes: "3"

plugins.security.nodes_dn:
- "CN=ccs-wazuh-indexer-1,OU=Wazuh,O=Wazuh,L=California,C=US" <-- CHANGE THIS -->
#- "CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US"
#- "CN=node-3,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

The lines <— CHANGE THIS — > has to be modified & need to enter the correct details we entered in the config.yaml file when deploying the certificates on the start

Replace the values of:

network.host with the IP address of the Wazuh indexer node.
node.name with the name of the Wazuh indexer node set in the config.yml file.
cluster.initial_master_nodes with the name of the Wazuh indexer node in the CCS environment.
cluster.name with the name of the cluster. For example, ccs-cluster.
plugins.security.nodes_dn with the distinguished name (DN) of the certificate of the CCS Wazuh indexer nodes. The common name (CN) is to be replaced with the node name of the CCS Wazuh indexer node provided in the config.yml file.

Run the following commands, replacing <INDEXER_NODE_NAME> with the value of node.name configured in the /etc/wazuh-indexer/opensearch.yml file:

NODE_NAME=ccs-wazuh-indexer-1 <-- CHANGE THIS -->
mkdir /etc/wazuh-indexer/certs
tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
chmod 500 /etc/wazuh-indexer/certs
chmod 400 /etc/wazuh-indexer/certs/*
chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

Enable and start the Wazuh indexer service:

systemctl daemon-reload
systemctl enable wazuh-indexer
systemctl start wazuh-indexer

Run the Wazuh indexer indexer-security-init.sh script on the Wazuh indexer node to load the new certificate information and initialize the Wazuh indexer:

/usr/share/wazuh-indexer/bin/indexer-security-init.sh

The output should look like :

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.168.186.60:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.10.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: ccs-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

DONE WITH SUCCESS & Cluster State Green defines the indexer is correctly configured

Check that the Wazuh indexer is initialized and properly set up by running the following command:

curl -k -u admin:admin https://192.168.146.154:9200

The output should look like :

{
  "name" : "ccs-wazuh-indexer-1",
  "cluster_name" : "ccs-cluster",
  "cluster_uuid" : "JkH8XuirS6m6PgdZFEQV2g",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "eee49cb340edc6c4d489bcd9324dda571fc8dc03",
    "build_date" : "2023-09-20T23:54:29.889267151Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Installing & Configuring Wazuh Dashboard

apt-get install debhelper tar curl libcap2-bin #debhelper version 9 or later
apt-get install gnupg apt-transport-https
## INSTALLING GPG KEY
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import && chmod 644 /usr/share/keyrings/wazuh.gpg
## ADDING THE REPOSITORY
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
## UPATING PACKAGES
apt-get update
apt-get -y install wazuh-dashboard

Edit the /etc/wazuh-dashboard/opensearch_dashboards.yml file to add the URL value of the Wazuh indexer in the opensearch.hosts section.

server.host: 0.0.0.0
   server.port: 443
   opensearch.hosts: "https://192.168.146.154:9200" <--CHANGE THIS-->
   opensearch.ssl.verificationMode: certificate

Replace <DASHBOARD_NODE_NAME> with your Wazuh dashboard node name, the same one set in config.yml when creating the certificates, and move the certificates to their corresponding directories:

NODE_NAME=ccs-wazuh-dashboard <--CHANGE THIS-->
mkdir /etc/wazuh-dashboard/certs
tar -xf ./wazuh-certificates.tar -C /etc/wazuh-dashboard/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem
mv -n /etc/wazuh-dashboard/certs/$NODE_NAME.pem /etc/wazuh-dashboard/certs/dashboard.pem
mv -n /etc/wazuh-dashboard/certs/$NODE_NAME-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem
chmod 500 /etc/wazuh-dashboard/certs
chmod 400 /etc/wazuh-dashboard/certs/*
chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs

Enable and start the Wazuh dashboard service:

systemctl daemon-reload
systemctl enable wazuh-dashboard
systemctl start wazuh-dashboard

Edit the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml file and replace the hosts section with a similar configuration as below:

hosts:
  - Cluster A:
      url: https://192.168.146.155 <--CHANGE THIS-->
      port: 55000
      username: wazuh-wui
      password: wazuh-wui
      run_as: true
  - Cluster B:
      url: https://192.168.146.156 <--CHANGE THIS-->
      port: 55000
      username: wazuh-wui
      password: wazuh-wui
      run_as: true

Restart the Wazuh dashboard service to apply the changes:

systemctl restart wazuh-dashboard

In the next section we will configure the Clusters & API connections

PreviousSingle node deployment with Script NextCluster Configuration

Last updated 2 months ago

Use Case:

Solution Overview:

CCS Environment: This environment is used only for querying data from remote Wazuh clusters. It contains the Wazuh indexer and dashboard but no agents or Wazuh server. It is trusted by clusters A and B.
Cluster A: Contains a Wazuh server and indexer for Organization A, with no dashboard (visualization occurs in the CCS environment). It trusts the CCS environment for data access.
Cluster B: Contains a Wazuh server and indexer for Organization B, also with no dashboard, but trusts the CCS environment for data access.

The table below shows a further breakdown of the infrastructure.

Cluster

Component

Node name

IP address

CCS

Wazuh dashboard

ccs-wazuh-dashboard

192.168.146.154

Wazuh indexer

ccs-wazuh-indexer-1

Cluster A

Wazuh server

ca-wazuh-server-1

192.168.146.155

Wazuh indexer

ca-wazuh-indexer-1

Cluster B

Wazuh server

cb-wazuh-server-1

192.168.146.156

Wazuh indexer

cb-wazuh-indexer-1

Ensure network connectivity between the remote clusters and the CCS environment, irrespective of the network address.

CSS Environment Configuration :

Generating certificates

# ENSURE YOU ARE CHANGING URLS BASED ON LATEST STABLE VERSION OF WAZUH
curl -sO https://packages.wazuh.com/4.9/config.yml
curl -sO https://packages.wazuh.com/4.9/wazuh-certs-tool.sh

Edit the config.yml file and replace the name and IP variables with the corresponding node names and IP addresses:

nodes:
  # Wazuh indexer nodes
  indexer:
    - name: ccs-wazuh-indexer-1
      ip: "192.168.146.154"
  # Wazuh dashboard nodes
  dashboard:
    - name: ccs-wazuh-dashboard
      ip: "192.168.146.154"

Note: Comment out or remove the Wazuh server configuration since it is not in use.

Run the wazuh-certs-tool.sh script with option -A to create the root-ca, admin, and node certificates:

bash ./wazuh-certs-tool.sh -A
# THE OUTPUT SHOULD LOOK LIKE 
25/08/2024 15:22:57 INFO: Generating the root certificate.
25/08/2024 15:22:57 INFO: Generating Admin certificates.
25/08/2024 15:22:58 INFO: Admin certificates created.
25/08/2024 15:22:58 INFO: Generating Wazuh indexer certificates.
25/08/2024 15:22:58 INFO: Wazuh indexer certificates created.
25/08/2024 15:22:58 INFO: Generating Wazuh dashboard certificates.
25/08/2024 15:22:59 INFO: Wazuh dashboard certificates created.

Copy the root CA certificate to the working directory of a node in cluster A and cluster B using the scp utility. The root CA is used later in the remote clusters to generate certificates.

Here we will pass the root certificates file into the nodes to ensure the connections are encrypted between master & worker nodes

scp -r wazuh-certificates/root-ca.* <USER_NAME>@<IP_ADDRESS>:<DESTINATION_DIRECTORY>
## MY COMMANDS
#scp -r wazuh-certificates/root-ca.* clustera@192.168.146.155:/home/clustera
#scp -r wazuh-certificates/root-ca.* clustern@192.168.146.156:/home/clusterb

Compress all the certificate files and remove the uncompressed version to allow for easier transfer to other component nodes within the CCS environment if need be:

tar -cvf ./wazuh-certificates.tar -C ./wazuh-certificates/ .

Installing & configuring Wazuh indexer

apt-get install debconf adduser procps
## ADDING WAZUH REPO
apt-get install gnupg apt-transport-httpscurl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import && chmod 644 /usr/share/keyrings/wazuh.gpg
## ADDING REPO SOURCE FILE
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
## UPDATING PACKAGES 
apt-get update

Installing Wazuh Indexer

apt-get -y install wazuh-indexer

Edit the /etc/wazuh-indexer/opensearch.yml configuration file:

network.host: "192.168.146.154" <-- CHANGE THIS -->
node.name: "ccs-wazuh-indexer-1" <-- CHANGE THIS ->

cluster.initial_master_nodes:
- "ccs-wazuh-indexer-1" <-- CHANGE THIS --> 
cluster.name: "ccs-cluster" <-- CHANGE THIS --> 
node.max_local_storage_nodes: "3"

plugins.security.nodes_dn:
- "CN=ccs-wazuh-indexer-1,OU=Wazuh,O=Wazuh,L=California,C=US" <-- CHANGE THIS -->
#- "CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US"
#- "CN=node-3,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

The lines <— CHANGE THIS — > has to be modified & need to enter the correct details we entered in the config.yaml file when deploying the certificates on the start

Replace the values of:

network.host with the IP address of the Wazuh indexer node.
node.name with the name of the Wazuh indexer node set in the config.yml file.
cluster.initial_master_nodes with the name of the Wazuh indexer node in the CCS environment.
cluster.name with the name of the cluster. For example, ccs-cluster.
plugins.security.nodes_dn with the distinguished name (DN) of the certificate of the CCS Wazuh indexer nodes. The common name (CN) is to be replaced with the node name of the CCS Wazuh indexer node provided in the config.yml file.

Run the following commands, replacing <INDEXER_NODE_NAME> with the value of node.name configured in the /etc/wazuh-indexer/opensearch.yml file:

NODE_NAME=ccs-wazuh-indexer-1 <-- CHANGE THIS -->
mkdir /etc/wazuh-indexer/certs
tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
chmod 500 /etc/wazuh-indexer/certs
chmod 400 /etc/wazuh-indexer/certs/*
chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

Enable and start the Wazuh indexer service:

systemctl daemon-reload
systemctl enable wazuh-indexer
systemctl start wazuh-indexer

Run the Wazuh indexer indexer-security-init.sh script on the Wazuh indexer node to load the new certificate information and initialize the Wazuh indexer:

/usr/share/wazuh-indexer/bin/indexer-security-init.sh

The output should look like :

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.168.186.60:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.10.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: ccs-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

DONE WITH SUCCESS & Cluster State Green defines the indexer is correctly configured

Check that the Wazuh indexer is initialized and properly set up by running the following command:

curl -k -u admin:admin https://192.168.146.154:9200

The output should look like :

{
  "name" : "ccs-wazuh-indexer-1",
  "cluster_name" : "ccs-cluster",
  "cluster_uuid" : "JkH8XuirS6m6PgdZFEQV2g",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "eee49cb340edc6c4d489bcd9324dda571fc8dc03",
    "build_date" : "2023-09-20T23:54:29.889267151Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Installing & Configuring Wazuh Dashboard

apt-get install debhelper tar curl libcap2-bin #debhelper version 9 or later
apt-get install gnupg apt-transport-https
## INSTALLING GPG KEY
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import && chmod 644 /usr/share/keyrings/wazuh.gpg
## ADDING THE REPOSITORY
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list
## UPATING PACKAGES
apt-get update
apt-get -y install wazuh-dashboard

Edit the /etc/wazuh-dashboard/opensearch_dashboards.yml file to add the URL value of the Wazuh indexer in the opensearch.hosts section.

server.host: 0.0.0.0
   server.port: 443
   opensearch.hosts: "https://192.168.146.154:9200" <--CHANGE THIS-->
   opensearch.ssl.verificationMode: certificate

Replace <DASHBOARD_NODE_NAME> with your Wazuh dashboard node name, the same one set in config.yml when creating the certificates, and move the certificates to their corresponding directories:

NODE_NAME=ccs-wazuh-dashboard <--CHANGE THIS-->
mkdir /etc/wazuh-dashboard/certs
tar -xf ./wazuh-certificates.tar -C /etc/wazuh-dashboard/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./root-ca.pem
mv -n /etc/wazuh-dashboard/certs/$NODE_NAME.pem /etc/wazuh-dashboard/certs/dashboard.pem
mv -n /etc/wazuh-dashboard/certs/$NODE_NAME-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem
chmod 500 /etc/wazuh-dashboard/certs
chmod 400 /etc/wazuh-dashboard/certs/*
chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs

Enable and start the Wazuh dashboard service:

systemctl daemon-reload
systemctl enable wazuh-dashboard
systemctl start wazuh-dashboard

Edit the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml file and replace the hosts section with a similar configuration as below:

hosts:
  - Cluster A:
      url: https://192.168.146.155 <--CHANGE THIS-->
      port: 55000
      username: wazuh-wui
      password: wazuh-wui
      run_as: true
  - Cluster B:
      url: https://192.168.146.156 <--CHANGE THIS-->
      port: 55000
      username: wazuh-wui
      password: wazuh-wui
      run_as: true

Restart the Wazuh dashboard service to apply the changes:

systemctl restart wazuh-dashboard

Note: Accessing the Wazuh dashboard at this configuration stage will produce API errors since the Wazuh servers have not been installed yet. Complete all the steps before accessing the dashboard in the section.

In the next section we will configure the Clusters & API connections