OpenSearch/Wazuh Indexer Setup and Management Guide#

This document provides comprehensive instructions for setting up, configuring, and managing an OpenSearch cluster that serves as a Wazuh indexer. It covers installation, backup procedures, configuration paths, and basic health checks for production environments.

System Overview#

The setup consists of:

OpenSearch 2.19.0 serving as a Wazuh indexer
Single-node cluster configuration (expandable to multi-node)
Security plugin enabled with admin authentication
Custom paths configured for Wazuh integration

Directory Structure#

The key directories and configuration files are:

1
/var/lib/wazuh-indexer/      # Main data directory
2
/var/log/wazuh-indexer/      # Log directory
3
/etc/opensearch/             # Configuration directory
4
/etc/opensearch/opensearch.yml  # Main configuration file
5
/etc/opensearch/certs/       # TLS certificates directory

Installation and Setup#

1. Install OpenSearch#

First, install the OpenSearch Debian package:

1
# Update package repository
2
sudo apt-get update
3

4
# Install required dependencies
5
sudo apt-get install -y curl gnupg2
6

7
# Add OpenSearch GPG key and repository
8
curl -o- https://artifacts.opensearch.org/publickeys/opensearch.pgp | sudo apt-key add -
9
echo "deb https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/opensearch-2.x.list
10

11
# Update repository and install OpenSearch
12
sudo apt-get update
13
sudo apt-get install opensearch=2.19.0

2. Configure Custom Paths#

Edit the OpenSearch configuration file to use Wazuh-specific paths:

1
sudo nano /etc/opensearch/opensearch.yml

Add or modify the following settings:

1
# Basic cluster configuration
2
cluster.name: wazuh-cluster
3
node.name: wazuh-indexer-1
4
network.host: 0.0.0.0
5
http.port: 9200
6
transport.port: 9300
7

8
# Custom paths for Wazuh integration
9
path.data: /var/lib/wazuh-indexer
10
path.logs: /var/log/wazuh-indexer
11
path.repo: /var/lib/wazuh-indexer/backup
12

13
# Memory settings
14
bootstrap.memory_lock: true
15

16
# Security plugin configuration
17
plugins.security.disabled: false
18
plugins.security.allow_default_init_securityindex: true

3. Set Proper Permissions#

Ensure the OpenSearch user has proper permissions to the data and log directories:

1
# Create directories
2
sudo mkdir -p /var/lib/wazuh-indexer
3
sudo mkdir -p /var/log/wazuh-indexer
4
sudo mkdir -p /var/lib/wazuh-indexer/backup
5

6
# Set ownership and permissions
7
sudo chown -R opensearch:opensearch /var/lib/wazuh-indexer
8
sudo chown -R opensearch:opensearch /var/log/wazuh-indexer
9
sudo chmod -R 755 /var/lib/wazuh-indexer
10
sudo chmod -R 755 /var/log/wazuh-indexer

4. Configure JVM Settings#

Set appropriate heap size for your system:

1
sudo nano /etc/opensearch/jvm.options

Modify heap settings (use 50% of available RAM, max 32GB):

1
# For 8GB system
2
-Xms4g
3
-Xmx4g
4

5
# For 16GB system
6
-Xms8g
7
-Xmx8g

5. Start the Service#

Enable and start the OpenSearch service:

1
sudo systemctl daemon-reload
2
sudo systemctl enable opensearch
3
sudo systemctl start opensearch
4

5
# Check service status
6
sudo systemctl status opensearch

Configuration Management#

Basic Configuration File#

Here’s a complete example configuration for a single-node Wazuh indexer:

1
# Cluster configuration
2
cluster.name: wazuh-cluster
3
node.name: wazuh-indexer-1
4
node.roles: [cluster_manager, data, ingest]
5

6
# Network configuration
7
network.host: 0.0.0.0
8
http.port: 9200
9
transport.port: 9300
10

11
# Discovery settings (single node)
12
discovery.type: single-node
13

14
# Paths
15
path.data: /var/lib/wazuh-indexer
16
path.logs: /var/log/wazuh-indexer
17
path.repo: /var/lib/wazuh-indexer/backup
18

19
# Memory
20
bootstrap.memory_lock: true
21

22
# Security plugin settings
23
plugins.security.disabled: false
24
plugins.security.allow_default_init_securityindex: true
25
plugins.security.allow_unsafe_democertificates: true
26
plugins.security.audit.type: internal_opensearch
27

28
# Performance settings
29
indices.memory.index_buffer_size: 10%
30
thread_pool.write.queue_size: 1000
31
thread_pool.search.queue_size: 1000
32

33
# Index management
34
action.auto_create_index: true
35
action.destructive_requires_name: true

Multi-Node Configuration#

For production environments, configure multiple nodes:

1
# Node 1 configuration
2
cluster.name: wazuh-cluster
3
node.name: wazuh-indexer-1
4
node.roles: [cluster_manager, data]
5
network.host: 0.0.0.0
6
discovery.seed_hosts: ["10.0.1.10", "10.0.1.11", "10.0.1.12"]
7
cluster.initial_cluster_manager_nodes: ["wazuh-indexer-1"]
8

9
# Node 2 configuration
10
cluster.name: wazuh-cluster
11
node.name: wazuh-indexer-2
12
node.roles: [data]
13
network.host: 0.0.0.0
14
discovery.seed_hosts: ["10.0.1.10", "10.0.1.11", "10.0.1.12"]
15
cluster.initial_cluster_manager_nodes: ["wazuh-indexer-1"]

Backup Procedures#

1. Prepare for Backup#

For production environments, ensure data consistency before backup:

1
# Flush in-memory changes to disk
2
curl -k -u admin:password -X POST "https://localhost:9200/_flush"
3

4
# Optionally, disable shard allocation during backup
5
curl -k -u admin:password -X PUT "https://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
6
{
7
  "persistent": {
8
    "cluster.routing.allocation.enable": "primaries"
9
  }
10
}'

2. Create Filesystem Backup#

Create a complete backup of the data directory:

1
# Stop OpenSearch service (for cold backup)
2
sudo systemctl stop opensearch
3

4
# Create a dated backup
5
sudo tar -czf /backup/wazuh-indexer-backup-$(date +%Y%m%d-%H%M%S).tar.gz \
6
  -C / var/lib/wazuh-indexer var/log/wazuh-indexer etc/opensearch
7

8
# Restart OpenSearch
9
sudo systemctl start opensearch

3. Create Snapshot Backup (Hot Backup)#

Configure and use OpenSearch snapshot feature:

1
# Register snapshot repository
2
curl -k -u admin:password -X PUT "https://localhost:9200/_snapshot/wazuh_backup" -H 'Content-Type: application/json' -d'
3
{
4
  "type": "fs",
5
  "settings": {
6
    "location": "/var/lib/wazuh-indexer/backup",
7
    "compress": true
8
  }
9
}'
10

11
# Create snapshot
12
curl -k -u admin:password -X PUT "https://localhost:9200/_snapshot/wazuh_backup/snapshot_$(date +%Y%m%d_%H%M%S)" -H 'Content-Type: application/json' -d'
13
{
14
  "indices": "wazuh-*",
15
  "ignore_unavailable": true,
16
  "include_global_state": false
17
}'
18

19
# Check snapshot status
20
curl -k -u admin:password "https://localhost:9200/_snapshot/wazuh_backup/_current"

4. Verify the Backup#

Always verify your backups:

1
# Check backup file size and integrity
2
du -sh /backup/wazuh-indexer-backup-*.tar.gz
3
tar -tzf /backup/wazuh-indexer-backup-*.tar.gz | head -20
4

5
# For snapshots, list all snapshots
6
curl -k -u admin:password "https://localhost:9200/_snapshot/wazuh_backup/_all?pretty"

5. Automated Backup Script#

Create an automated backup script:

1
#!/bin/bash
2
BACKUP_DIR="/backup/wazuh-indexer"
3
DATE=$(date +%Y%m%d-%H%M%S)
4
RETENTION_DAYS=30
5

6
# Create backup directory
7
mkdir -p $BACKUP_DIR
8

9
# Create snapshot backup
10
curl -k -u admin:password -X PUT "https://localhost:9200/_snapshot/wazuh_backup/auto_snapshot_$DATE" \
11
  -H 'Content-Type: application/json' -d'
12
{
13
  "indices": "wazuh-*",
14
  "ignore_unavailable": true,
15
  "include_global_state": false
16
}'
17

18
# Wait for snapshot completion
19
while true; do
20
  STATUS=$(curl -s -k -u admin:password "https://localhost:9200/_snapshot/wazuh_backup/auto_snapshot_$DATE" | jq -r '.snapshots[0].state')
21
  if [ "$STATUS" = "SUCCESS" ]; then
22
    echo "Snapshot completed successfully"
23
    break
24
  elif [ "$STATUS" = "FAILED" ]; then
25
    echo "Snapshot failed"
26
    exit 1
27
  fi
28
  sleep 10
29
done
30

31
# Clean up old snapshots
32
find $BACKUP_DIR -name "auto_snapshot_*" -mtime +$RETENTION_DAYS -delete
33

34
# Log backup completion
35
echo "$(date): Backup completed successfully" >> /var/log/wazuh-backup.log

Health Checks and Monitoring#

1. Basic Health Checks#

Regularly monitor the health of your OpenSearch/Wazuh indexer cluster:

1
# Check cluster health
2
curl -k -u admin:password "https://localhost:9200/_cluster/health?pretty"
3

4
# Expected output shows cluster status (green or yellow for single-node deployments)

2. Node Status and Performance#

1
# Check node information
2
curl -k -u admin:password "https://localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent"
3

4
# Check indices status
5
curl -k -u admin:password "https://localhost:9200/_cat/indices?v&s=index"
6

7
# Check shard allocation
8
curl -k -u admin:password "https://localhost:9200/_cat/shards?v"

3. Performance Metrics#

1
# Check cluster statistics
2
curl -k -u admin:password "https://localhost:9200/_cluster/stats?pretty"
3

4
# Check thread pool status
5
curl -k -u admin:password "https://localhost:9200/_cat/thread_pool?v&h=name,active,queue,rejected"
6

7
# Monitor search and indexing performance
8
curl -k -u admin:password "https://localhost:9200/_stats/search,indexing?pretty"

4. Wazuh-Specific Monitoring#

1
# Check Wazuh indices
2
curl -k -u admin:password "https://localhost:9200/_cat/indices/wazuh-*?v&s=index"
3

4
# Check latest Wazuh alerts
5
curl -k -u admin:password "https://localhost:9200/wazuh-alerts-*/_search?size=5&sort=@timestamp:desc&pretty"
6

7
# Monitor Wazuh data ingestion rate
8
curl -k -u admin:password "https://localhost:9200/wazuh-alerts-*/_count?pretty"

5. Automated Health Check Script#

1
#!/bin/bash
2
# Colors for output
3
RED='\033[0;31m'
4
GREEN='\033[0;32m'
5
YELLOW='\033[1;33m'
6
NC='\033[0m' # No Color
7

8
echo "=== Wazuh Indexer Health Check ==="
9

10
# Check if OpenSearch is running
11
if ! systemctl is-active --quiet opensearch; then
12
    echo -e "${RED}ERROR: OpenSearch service is not running${NC}"
13
    exit 1
14
fi
15

16
# Check cluster health
17
HEALTH=$(curl -s -k -u admin:password "https://localhost:9200/_cluster/health" | jq -r '.status')
18
case $HEALTH in
19
    "green")
20
        echo -e "${GREEN}✓ Cluster health: $HEALTH${NC}"
21
        ;;
22
    "yellow")
23
        echo -e "${YELLOW}⚠ Cluster health: $HEALTH${NC}"
24
        ;;
25
    "red")
26
        echo -e "${RED}✗ Cluster health: $HEALTH${NC}"
27
        ;;
28
esac
29

30
# Check disk usage
31
DISK_USAGE=$(df -h /var/lib/wazuh-indexer | awk 'NR==2 {print $5}' | sed 's/%//')
32
if [ $DISK_USAGE -gt 80 ]; then
33
    echo -e "${RED}✗ High disk usage: ${DISK_USAGE}%${NC}"
34
else
35
    echo -e "${GREEN}✓ Disk usage: ${DISK_USAGE}%${NC}"
36
fi
37

38
# Check memory usage
39
HEAP_USAGE=$(curl -s -k -u admin:password "https://localhost:9200/_cat/nodes?h=heap.percent" | tr -d ' ')
40
if [ $HEAP_USAGE -gt 85 ]; then
41
    echo -e "${RED}✗ High heap usage: ${HEAP_USAGE}%${NC}"
42
else
43
    echo -e "${GREEN}✓ Heap usage: ${HEAP_USAGE}%${NC}"
44
fi
45

46
echo "=== Health Check Complete ==="

Troubleshooting#

Common Issues and Solutions#

1. Yellow Cluster Status#

Issue: Single-node cluster shows yellow status due to unassigned replica shards.

Solution: This is normal for single-node deployments. For production, either:

1
# Option 1: Add more nodes to the cluster
2
# Option 2: Set replica count to 0 for single node
3
curl -k -u admin:password -X PUT "https://localhost:9200/_template/wazuh" -H 'Content-Type: application/json' -d'
4
{
5
  "index_patterns": ["wazuh-*"],
6
  "settings": {
7
    "number_of_replicas": 0
8
  }
9
}'

2. High Memory Usage#

Issue: OpenSearch consumes too much memory.

Solution:

1
# Adjust JVM heap settings
2
sudo nano /etc/opensearch/jvm.options
3

4
# Set heap to 50% of available RAM (max 32GB)
5
-Xms4g
6
-Xmx4g
7

8
# Clear fielddata cache
9
curl -k -u admin:password -X POST "https://localhost:9200/_cache/clear?fielddata=true"

3. Disk Space Issues#

Issue: Running out of disk space.

Solution:

1
# Check index sizes
2
curl -k -u admin:password "https://localhost:9200/_cat/indices?v&s=store.size:desc"
3

4
# Delete old indices
5
curl -k -u admin:password -X DELETE "https://localhost:9200/wazuh-alerts-2023.01.*"
6

7
# Set up index lifecycle management
8
curl -k -u admin:password -X PUT "https://localhost:9200/_template/wazuh-alerts" -H 'Content-Type: application/json' -d'
9
{
10
  "index_patterns": ["wazuh-alerts-*"],
11
  "settings": {
12
    "index.lifecycle.name": "wazuh-policy",
13
    "index.lifecycle.rollover_alias": "wazuh-alerts"
14
  }
15
}'

4. Authentication Issues#

Issue: Cannot authenticate with admin credentials.

Solution:

1
# Reset admin password
2
sudo /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
3
  -cd /etc/opensearch/opensearch-security/ \
4
  -icl -nhnv \
5
  -cacert /etc/opensearch/certs/root-ca.pem \
6
  -cert /etc/opensearch/certs/admin.pem \
7
  -key /etc/opensearch/certs/admin-key.pem
8

9
# Update internal users
10
sudo nano /etc/opensearch/opensearch-security/internal_users.yml

Maintenance Procedures#

Regular Maintenance Tasks#

1. Index Management#

1
# Force merge old indices (monthly)
2
curl -k -u admin:password -X POST "https://localhost:9200/wazuh-alerts-2023.12.*/_forcemerge?max_num_segments=1"
3

4
# Update index settings for better performance
5
curl -k -u admin:password -X PUT "https://localhost:9200/wazuh-alerts-*/_settings" -H 'Content-Type: application/json' -d'
6
{
7
  "index": {
8
    "refresh_interval": "30s",
9
    "number_of_replicas": 0
10
  }
11
}'

2. Performance Optimization#

1
# Clear caches
2
curl -k -u admin:password -X POST "https://localhost:9200/_cache/clear"
3

4
# Optimize indices
5
curl -k -u admin:password -X POST "https://localhost:9200/_optimize?max_num_segments=1"
6

7
# Check slow queries
8
curl -k -u admin:password "https://localhost:9200/_nodes/stats/indices/search?pretty" | jq '.nodes[].indices.search'

Complete Removal#

To completely remove OpenSearch from the system:

1
# Stop the service
2
sudo systemctl stop opensearch
3
sudo systemctl disable opensearch
4

5
# Remove the package
6
sudo apt-get remove --purge opensearch
7

8
# Remove configuration directories
9
sudo rm -rf /etc/opensearch
10

11
# Remove data directories
12
sudo rm -rf /var/lib/opensearch
13
sudo rm -rf /var/lib/wazuh-indexer
14
sudo rm -rf /var/log/opensearch
15
sudo rm -rf /var/log/wazuh-indexer
16

17
# Remove the user
18
sudo userdel opensearch
19

20
# Remove systemd files
21
sudo rm -f /etc/systemd/system/opensearch.service
22
sudo systemctl daemon-reload
23

24
# Remove repository
25
sudo rm -f /etc/apt/sources.list.d/opensearch-2.x.list
26

27
# Check for any remaining files
28
sudo find / -name "*opensearch*" -type d 2>/dev/null
29
sudo find / -name "*wazuh-indexer*" -type d 2>/dev/null

Security Considerations#

Production Security Checklist#

Change Default Passwords: Never use default credentials in production
Enable TLS: Configure proper TLS certificates for all communications
Network Security: Use firewall rules to restrict access to OpenSearch ports
User Management: Implement proper role-based access control
Audit Logging: Enable and monitor security audit logs
Regular Updates: Keep OpenSearch updated to address security vulnerabilities
Backup Security: Encrypt backup files and secure backup storage

Security Configuration Example#

1
# Security settings in opensearch.yml
2
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem
3
plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem
4
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
5
plugins.security.ssl.transport.enforce_hostname_verification: false
6

7
plugins.security.ssl.http.enabled: true
8
plugins.security.ssl.http.pemcert_filepath: certs/node.pem
9
plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem
10
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem
11

12
plugins.security.authcz.admin_dn:
13
  - CN=admin,OU=client,O=client,L=Test,C=DE
14

15
plugins.security.audit.type: internal_opensearch
16
plugins.security.audit.config.disabled_rest_categories: NONE
17
plugins.security.audit.config.disabled_transport_categories: NONE

Production Recommendations#

For production environments:

Use Multiple Nodes: Deploy at least 3 nodes for high availability
Separate Roles: Use dedicated master nodes for large clusters
Monitor Resources: Implement comprehensive monitoring with Prometheus/Grafana
Backup Strategy: Implement automated daily backups with off-site storage
Index Lifecycle: Configure automatic index rotation and deletion
Load Balancing: Use a load balancer for client connections
Capacity Planning: Monitor growth trends and plan for capacity expansion

This guide provides a solid foundation for deploying and managing OpenSearch as a Wazuh indexer in both development and production environments.