Elasticsearch and OpenSearch Cluster Health Troubleshooting Guide#

Maintaining healthy Elasticsearch and OpenSearch clusters is crucial for any SIEM or log analytics infrastructure. This comprehensive guide covers common cluster health issues, their diagnosis, and secure resolution methods.

Table of Contents#

Initial Assessment
Common Issues and Solutions
Security Considerations
Preventive Measures
Monitoring and Alerting

Initial Assessment#

Before implementing any fixes, perform a thorough assessment of your cluster’s current state.

Basic Health Check#

1
# Check overall cluster health
2
curl -k -X GET "https://<elasticsearch-host>:9200/_cluster/health?pretty" \
3
     -u <username>:<password>
4

5
# Expected output format:
6
{
7
  "cluster_name" : "your-cluster",
8
  "status" : "yellow",
9
  "timed_out" : false,
10
  "number_of_nodes" : 1,
11
  "number_of_data_nodes" : 1,
12
  "active_primary_shards" : 15,
13
  "active_shards" : 15,
14
  "relocating_shards" : 0,
15
  "initializing_shards" : 0,
16
  "unassigned_shards" : 5,
17
  "delayed_unassigned_shards" : 0,
18
  "number_of_pending_tasks" : 0,
19
  "number_of_in_flight_fetch" : 0,
20
  "task_max_waiting_in_queue_millis" : 0,
21
  "active_shards_percent_as_number" : 75.0
22
}

Detailed Shard Analysis#

1
# Get detailed shard allocation explanation
2
curl -k -X GET "https://<elasticsearch-host>:9200/_cluster/allocation/explain?pretty" \
3
     -u <username>:<password>
4

5
# List indices with detailed health status
6
curl -k -X GET "https://<elasticsearch-host>:9200/_cat/indices?v&h=index,health,status,pri,rep,docs.count,store.size,unassign.shards" \
7
     -u <username>:<password>
8

9
# Check specific shard allocation
10
curl -k -X GET "https://<elasticsearch-host>:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason" \
11
     -u <username>:<password>

Node Information#

1
# Check node information and resources
2
curl -k -X GET "https://<elasticsearch-host>:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent,node.role,master" \
3
     -u <username>:<password>
4

5
# Get cluster settings
6
curl -k -X GET "https://<elasticsearch-host>:9200/_cluster/settings?pretty" \
7
     -u <username>:<password>

Common Issues and Solutions#

1. Yellow Cluster Status with Unassigned Replicas (Single Node)#

Problem: In single-node clusters, replica shards cannot be allocated, causing yellow status.

Root Cause: Elasticsearch/OpenSearch cannot place replica shards on the same node as primary shards for redundancy.

Solution: Disable replicas for single-node deployments.

1
# Disable replicas for all existing indices
2
curl -k -X PUT "https://<elasticsearch-host>:9200/*/_settings" \
3
     -H "Content-Type: application/json" \
4
     -u <username>:<password> \
5
     -d '{
6
       "index": {
7
         "number_of_replicas": 0
8
       }
9
     }'
10

11
# Set template for future indices
12
curl -k -X PUT "https://<elasticsearch-host>:9200/_template/default_template" \
13
     -H "Content-Type: application/json" \
14
     -u <username>:<password> \
15
     -d '{
16
       "index_patterns": ["*"],
17
       "settings": {
18
         "number_of_replicas": 0
19
       },
20
       "order": 1
21
     }'
22

23
# Verify template creation
24
curl -k -X GET "https://<elasticsearch-host>:9200/_template/default_template?pretty" \
25
     -u <username>:<password>

2. OpenDistro/ISM Index Issues#

Problem: OpenDistro Index State Management indices showing yellow status.

Root Cause: ISM indices created with default replica settings.

Solution: Update ISM-specific indices and create dedicated templates.

1
# Update existing ISM indices
2
curl -k -X PUT "https://<elasticsearch-host>:9200/.opendistro-ism*/_settings" \
3
     -H "Content-Type: application/json" \
4
     -u <username>:<password> \
5
     -d '{
6
       "index": {
7
         "number_of_replicas": 0
8
       }
9
     }'
10

11
# Create ISM-specific template
12
curl -k -X PUT "https://<elasticsearch-host>:9200/_template/ism_template" \
13
     -H "Content-Type: application/json" \
14
     -u <username>:<password> \
15
     -d '{
16
       "index_patterns": [".opendistro-ism*", ".opensearch-ism*"],
17
       "settings": {
18
         "number_of_replicas": 0,
19
         "index": {
20
           "auto_expand_replicas": "0-1"
21
         }
22
       },
23
       "order": 10
24
     }'
25

26
# Update OpenSearch security indices
27
curl -k -X PUT "https://<elasticsearch-host>:9200/.opensearch-*/_settings" \
28
     -H "Content-Type: application/json" \
29
     -u <username>:<password> \
30
     -d '{
31
       "index": {
32
         "number_of_replicas": 0
33
       }
34
     }'

3. Monitoring and Security Index Issues#

Problem: Wazuh, Filebeat, or other monitoring indices showing unassigned shards.

Root Cause: Default templates creating replicas in single-node environments.

Solution: Create comprehensive monitoring templates.

1
# Create monitoring indices template
2
curl -k -X PUT "https://<elasticsearch-host>:9200/_template/monitoring_template" \
3
     -H "Content-Type: application/json" \
4
     -u <username>:<password> \
5
     -d '{
6
       "index_patterns": [
7
         "wazuh-*",
8
         "filebeat-*",
9
         "otel-v1-*",
10
         "logstash-*",
11
         "metricbeat-*",
12
         "winlogbeat-*"
13
       ],
14
       "settings": {
15
         "number_of_replicas": 0,
16
         "index": {
17
           "lifecycle": {
18
             "name": "monitoring_policy",
19
             "rollover_alias": "monitoring"
20
           }
21
         }
22
       },
23
       "order": 5
24
     }'
25

26
# Update existing monitoring indices
27
indices=("wazuh-*" "filebeat-*" "otel-v1-*" "logstash-*")
28
for pattern in "${indices[@]}"; do
29
  curl -k -X PUT "https://<elasticsearch-host>:9200/${pattern}/_settings" \
30
       -H "Content-Type: application/json" \
31
       -u <username>:<password> \
32
       -d '{
33
         "index": {
34
           "number_of_replicas": 0
35
         }
36
       }'
37
done

4. Disk Space Issues#

Problem: Cluster showing red status due to disk space constraints.

Solution: Implement disk management and cleanup procedures.

1
# Check disk usage by index
2
curl -k -X GET "https://<elasticsearch-host>:9200/_cat/indices?v&s=store.size:desc&h=index,store.size,docs.count" \
3
     -u <username>:<password>
4

5
# Configure disk watermarks
6
curl -k -X PUT "https://<elasticsearch-host>:9200/_cluster/settings" \
7
     -H "Content-Type: application/json" \
8
     -u <username>:<password> \
9
     -d '{
10
       "persistent": {
11
         "cluster.routing.allocation.disk.watermark.low": "85%",
12
         "cluster.routing.allocation.disk.watermark.high": "90%",
13
         "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
14
         "cluster.routing.allocation.disk.include_relocations": false
15
       }
16
     }'
17

18
# Delete old indices (be very careful with this command)
19
# List indices older than 30 days
20
curl -k -X GET "https://<elasticsearch-host>:9200/_cat/indices?v&s=creation.date" \
21
     -u <username>:<password>
22

23
# Example: Delete specific old index (replace with actual index name)
24
# curl -k -X DELETE "https://<elasticsearch-host>:9200/old-index-name" \
25
#      -u <username>:<password>

5. Shard Allocation Issues#

Problem: Shards stuck in initializing or relocating state.

Solution: Force shard allocation and resolve allocation issues.

1
# Check allocation filters
2
curl -k -X GET "https://<elasticsearch-host>:9200/_cluster/settings?pretty&include_defaults=true" \
3
     -u <username>:<password>
4

5
# Clear allocation filters if present
6
curl -k -X PUT "https://<elasticsearch-host>:9200/_cluster/settings" \
7
     -H "Content-Type: application/json" \
8
     -u <username>:<password> \
9
     -d '{
10
       "persistent": {
11
         "cluster.routing.allocation.include._name": null,
12
         "cluster.routing.allocation.exclude._name": null,
13
         "cluster.routing.allocation.require._name": null
14
       }
15
     }'
16

17
# Force allocation of unassigned shards
18
curl -k -X POST "https://<elasticsearch-host>:9200/_cluster/reroute?retry_failed=true" \
19
     -H "Content-Type: application/json" \
20
     -u <username>:<password> \
21
     -d '{
22
       "commands": [
23
         {
24
           "allocate_empty_primary": {
25
             "index": "problem-index",
26
             "shard": 0,
27
             "node": "node-name",
28
             "accept_data_loss": true
29
           }
30
         }
31
       ]
32
     }'

Post-Fix Verification#

After implementing fixes, verify the cluster health:

1
# Force cluster routing refresh
2
curl -k -X POST "https://<elasticsearch-host>:9200/_cluster/reroute?retry_failed=true" \
3
     -u <username>:<password>
4

5
# Wait a few moments and check health
6
sleep 30
7
curl -k -X GET "https://<elasticsearch-host>:9200/_cluster/health?pretty" \
8
     -u <username>:<password>
9

10
# Verify no unassigned shards
11
curl -k -X GET "https://<elasticsearch-host>:9200/_cat/shards?v&h=index,shard,prirep,state" \
12
     -u <username>:<password> | grep UNASSIGNED
13

14
# Check cluster stats
15
curl -k -X GET "https://<elasticsearch-host>:9200/_cluster/stats?pretty" \
16
     -u <username>:<password>

Security Considerations#

Certificate Management#

Replace -k flag with proper certificate validation:

1
# Use proper certificate validation
2
curl --cacert /path/to/ca.crt \
3
     -X GET "https://<elasticsearch-host>:9200/_cluster/health" \
4
     -u <username>:<password>
5

6
# For client certificates
7
curl --cert /path/to/client.crt \
8
     --key /path/to/client.key \
9
     --cacert /path/to/ca.crt \
10
     -X GET "https://<elasticsearch-host>:9200/_cluster/health"

Credential Management#

1
# Use environment variables for credentials
2
export ES_USERNAME="admin"
3
export ES_PASSWORD="your-secure-password"
4
export ES_HOST="https://elasticsearch:9200"
5

6
# Create a secure wrapper script
7
cat > es_health_check.sh << 'EOF'
8
#!/bin/bash
9
set -euo pipefail
10

11
# Load credentials from secure location
12
source /etc/elasticsearch/credentials
13

14
curl --cacert /etc/elasticsearch/certs/ca.crt \
15
     -X GET "${ES_HOST}/_cluster/health?pretty" \
16
     -u "${ES_USERNAME}:${ES_PASSWORD}"
17
EOF
18

19
chmod 700 es_health_check.sh

RBAC Implementation#

1
# Create a dedicated monitoring role
2
curl -k -X PUT "https://<elasticsearch-host>:9200/_security/role/cluster_monitor" \
3
     -H "Content-Type: application/json" \
4
     -u <username>:<password> \
5
     -d '{
6
       "cluster": ["monitor", "manage"],
7
       "indices": [
8
         {
9
           "names": ["*"],
10
           "privileges": ["monitor"]
11
         }
12
       ]
13
     }'
14

15
# Create monitoring user
16
curl -k -X PUT "https://<elasticsearch-host>:9200/_security/user/monitor_user" \
17
     -H "Content-Type: application/json" \
18
     -u <username>:<password> \
19
     -d '{
20
       "password": "secure-monitor-password",
21
       "roles": ["cluster_monitor"],
22
       "full_name": "Cluster Monitor User"
23
     }'

Network Security#

1
# Implement firewall rules
2
sudo ufw allow from 192.168.1.0/24 to any port 9200
3
sudo ufw allow from 192.168.1.0/24 to any port 9300
4

5
# Configure elasticsearch.yml for network binding
6
echo "network.host: 192.168.1.100" >> /etc/elasticsearch/elasticsearch.yml
7
echo "discovery.seed_hosts: [\"192.168.1.100\", \"192.168.1.101\"]" >> /etc/elasticsearch/elasticsearch.yml

Preventive Measures#

Monitoring Setup#

1
# Configure cluster-level monitoring
2
curl -k -X PUT "https://<elasticsearch-host>:9200/_cluster/settings" \
3
     -H "Content-Type: application/json" \
4
     -u <username>:<password> \
5
     -d '{
6
       "persistent": {
7
         "cluster.routing.allocation.cluster_concurrent_rebalance": 2,
8
         "cluster.routing.allocation.node_concurrent_recoveries": 2,
9
         "cluster.routing.allocation.node_initial_primaries_recoveries": 4
10
       }
11
     }'
12

13
# Set up index lifecycle management
14
curl -k -X PUT "https://<elasticsearch-host>:9200/_ilm/policy/monitoring_policy" \
15
     -H "Content-Type: application/json" \
16
     -u <username>:<password> \
17
     -d '{
18
       "policy": {
19
         "phases": {
20
           "hot": {
21
             "actions": {
22
               "rollover": {
23
                 "max_size": "10GB",
24
                 "max_age": "7d"
25
               }
26
             }
27
           },
28
           "warm": {
29
             "min_age": "7d",
30
             "actions": {
31
               "shrink": {
32
                 "number_of_shards": 1
33
               }
34
             }
35
           },
36
           "cold": {
37
             "min_age": "30d",
38
             "actions": {
39
               "readonly": {}
40
             }
41
           },
42
           "delete": {
43
             "min_age": "90d",
44
             "actions": {
45
               "delete": {}
46
             }
47
           }
48
         }
49
       }
50
     }'

Backup Strategy#

1
# Configure snapshot repository
2
curl -k -X PUT "https://<elasticsearch-host>:9200/_snapshot/backup_repo" \
3
     -H "Content-Type: application/json" \
4
     -u <username>:<password> \
5
     -d '{
6
       "type": "fs",
7
       "settings": {
8
         "location": "/backup/elasticsearch",
9
         "compress": true,
10
         "chunk_size": "100MB"
11
       }
12
     }'
13

14
# Create automated backup script
15
cat > backup_cluster.sh << 'EOF'
16
#!/bin/bash
17
set -euo pipefail
18

19
DATE=$(date +%Y%m%d_%H%M%S)
20
SNAPSHOT_NAME="backup_${DATE}"
21

22
curl -k -X PUT "https://elasticsearch:9200/_snapshot/backup_repo/${SNAPSHOT_NAME}?wait_for_completion=true" \
23
     -H "Content-Type: application/json" \
24
     -u "${ES_USERNAME}:${ES_PASSWORD}" \
25
     -d '{
26
       "indices": "*",
27
       "include_global_state": true,
28
       "ignore_unavailable": true
29
     }'
30

31
echo "Backup completed: ${SNAPSHOT_NAME}"
32
EOF
33

34
chmod +x backup_cluster.sh

Health Monitoring Script#

1
# Create comprehensive health monitoring script
2
cat > cluster_health_monitor.sh << 'EOF'
3
#!/bin/bash
4
set -euo pipefail
5

6
# Configuration
7
ES_HOST="${ES_HOST:-https://localhost:9200}"
8
ES_USER="${ES_USER:-admin}"
9
ES_PASS="${ES_PASS:-password}"
10
ALERT_EMAIL="${ALERT_EMAIL:-admin@company.com}"
11

12
# Colors for output
13
RED='\033[0;31m'
14
YELLOW='\033[1;33m'
15
GREEN='\033[0;32m'
16
NC='\033[0m'
17

18
# Check cluster health
19
health_status=$(curl -s -k -u "${ES_USER}:${ES_PASS}" \
20
  "${ES_HOST}/_cluster/health" | jq -r '.status')
21

22
case $health_status in
23
  "green")
24
    echo -e "${GREEN}✓ Cluster health: $health_status${NC}"
25
    ;;
26
  "yellow")
27
    echo -e "${YELLOW}⚠ Cluster health: $health_status${NC}"
28
    # Add notification logic here
29
    ;;
30
  "red")
31
    echo -e "${RED}✗ Cluster health: $health_status${NC}"
32
    # Add critical alert logic here
33
    ;;
34
esac
35

36
# Check disk usage
37
disk_usage=$(df -h /var/lib/elasticsearch | awk 'NR==2 {print $5}' | sed 's/%//')
38
if [ "$disk_usage" -gt 80 ]; then
39
  echo -e "${RED}✗ High disk usage: ${disk_usage}%${NC}"
40
else
41
  echo -e "${GREEN}✓ Disk usage: ${disk_usage}%${NC}"
42
fi
43

44
# Check unassigned shards
45
unassigned=$(curl -s -k -u "${ES_USER}:${ES_PASS}" \
46
  "${ES_HOST}/_cluster/health" | jq -r '.unassigned_shards')
47

48
if [ "$unassigned" -gt 0 ]; then
49
  echo -e "${YELLOW}⚠ Unassigned shards: $unassigned${NC}"
50
else
51
  echo -e "${GREEN}✓ All shards assigned${NC}"
52
fi
53

54
# Check node status
55
nodes=$(curl -s -k -u "${ES_USER}:${ES_PASS}" \
56
  "${ES_HOST}/_cat/nodes?h=name,heap.percent,disk.used_percent" | \
57
  while read name heap disk; do
58
    if [ "${heap%.*}" -gt 85 ]; then
59
      echo -e "${RED}✗ Node $name high heap: $heap${NC}"
60
    elif [ "${disk%.*}" -gt 85 ]; then
61
      echo -e "${YELLOW}⚠ Node $name high disk: $disk${NC}"
62
    else
63
      echo -e "${GREEN}✓ Node $name healthy${NC}"
64
    fi
65
  done)
66

67
echo "$nodes"
68
EOF
69

70
chmod +x cluster_health_monitor.sh

Monitoring and Alerting#

Prometheus Integration#

1
# prometheus.yml excerpt
2
scrape_configs:
3
  - job_name: "elasticsearch"
4
    static_configs:
5
      - targets: ["elasticsearch:9200"]
6
    metrics_path: "/_prometheus/metrics"
7
    basic_auth:
8
      username: "monitor_user"
9
      password: "secure-password"

Grafana Dashboard#

Key metrics to monitor:

Cluster health status
Node availability
Heap usage percentage
Disk usage percentage
Unassigned shard count
Search and indexing rates
JVM garbage collection metrics

Alerting Rules#

1
groups:
2
  - name: elasticsearch
3
    rules:
4
      - alert: ElasticsearchClusterYellow
5
        expr: elasticsearch_cluster_health_status{color="yellow"} == 1
6
        for: 5m
7
        labels:
8
          severity: warning
9
        annotations:
10
          summary: "Elasticsearch cluster status is yellow"
11

12
      - alert: ElasticsearchClusterRed
13
        expr: elasticsearch_cluster_health_status{color="red"} == 1
14
        for: 1m
15
        labels:
16
          severity: critical
17
        annotations:
18
          summary: "Elasticsearch cluster status is red"
19

20
      - alert: ElasticsearchHighHeapUsage
21
        expr: elasticsearch_jvm_memory_used_bytes{area="heap"} / elasticsearch_jvm_memory_max_bytes{area="heap"} > 0.85
22
        for: 5m
23
        labels:
24
          severity: warning
25
        annotations:
26
          summary: "Elasticsearch node heap usage is high"

Best Practices#

Regular Health Checks: Implement automated monitoring with appropriate alerting thresholds
Capacity Planning: Monitor trends and plan for capacity expansion before issues occur
Index Lifecycle Management: Implement proper ILM policies to manage data retention
Security Updates: Keep clusters updated with security patches
Backup Verification: Regularly test backup and restore procedures
Documentation: Maintain runbooks for common issues and procedures

Conclusion#

Maintaining healthy Elasticsearch and OpenSearch clusters requires proactive monitoring, proper configuration, and systematic troubleshooting approaches. This guide provides the foundation for identifying and resolving common cluster health issues while maintaining security best practices.

Remember to:

Always backup configurations before making changes
Test fixes in staging environments first
Monitor cluster health continuously
Implement proper security measures
Document all changes and procedures

Regular maintenance and monitoring will help prevent most cluster health issues and ensure optimal performance for your SIEM and analytics infrastructure.