OpenSearch Cluster Configuration: Master and Data Node Setup Guide#

This comprehensive guide covers the configuration of a production-ready OpenSearch cluster with proper node roles, security settings, and performance optimizations. The configuration shown here demonstrates a three-node cluster setup with one master node and two data nodes.

Overview#

OpenSearch is a powerful, open-source search and analytics engine that evolved from Elasticsearch. Setting up a properly configured cluster is essential for high availability, performance, and security in production environments.

Basic Cluster Configuration#

Master Node Configuration (os1)#

The master node is responsible for cluster management, node coordination, and maintaining cluster state. Here’s the configuration for the primary master node:

1
# Basic cluster configuration
2
cluster.name: opensearch-cluster
3
node.name: os1
4
node.roles: [cluster_manager, data]
5
network.host: 0.0.0.0
6
http.port: 9200
7
transport.port: 9300
8

9
# Discovery settings
10
discovery.seed_hosts:
11
  ["172.17.14.79:9300", "172.17.14.89:9300", "172.17.14.39:9300"]
12
cluster.initial_master_nodes: ["os1"]
13

14
# Repo for Migration
15
path.repo: /var/lib/opensearch/migration
16

17
# Memory and path settings
18
bootstrap.memory_lock: true
19
path.data: /var/lib/opensearch
20
path.logs: /var/log/opensearch
21

22
# Performance settings
23
indices.memory.index_buffer_size: 5%
24
thread_pool.write.queue_size: 200
25
thread_pool.search.queue_size: 500

Data Node Configuration (os2 & os3)#

Data nodes handle the actual storage and search operations. Here’s the configuration for data nodes:

1
# Basic cluster configuration
2
cluster.name: opensearch-cluster
3
node.name: os2 # Change to os3 for the third node
4
node.roles: [data, ingest]
5
network.host: 0.0.0.0
6
http.port: 9200
7
transport.port: 9300
8

9
# Discovery settings
10
discovery.seed_hosts:
11
  ["172.17.14.79:9300", "172.17.14.89:9300", "172.17.14.39:9300"]
12
cluster.initial_master_nodes: ["os1"]
13

14
# Repo for opensearch migration
15
path.repo: /var/lib/opensearch/migration
16

17
# Memory and path settings
18
bootstrap.memory_lock: true
19
path.data: /var/lib/opensearch
20
path.logs: /var/log/opensearch
21

22
# Performance settings
23
indices.memory.index_buffer_size: 5%
24
thread_pool.write.queue_size: 200
25
thread_pool.search.queue_size: 500
26

27
# Node Configuration
28
node.max_local_storage_nodes: 3

Security Configuration#

TLS/SSL Configuration#

Security is paramount in production OpenSearch deployments. The following configuration enables TLS encryption for both transport and HTTP layers:

1
# Security Configuration
2
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem
3
plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem
4
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
5
plugins.security.ssl.transport.enforce_hostname_verification: false
6
plugins.security.ssl.transport.resolve_hostname: false
7

8
plugins.security.ssl.http.enabled: true
9
plugins.security.ssl.http.pemcert_filepath: certs/node.pem
10
plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem
11
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem
12

13
plugins.security.allow_unsafe_democertificates: false
14
plugins.security.allow_default_init_securityindex: true

Node Authentication#

Proper node authentication ensures only authorized nodes can join the cluster:

1
# List all nodes DN (Distinguished Names)
2
plugins.security.nodes_dn:
3
  - "CN=opensearch-1,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
4
  - "CN=opensearch-2,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
5
  - "CN=opensearch-3,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
6

7
plugins.security.authcz.admin_dn:
8
  - "CN=admin,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
9
  - "CN=opensearch-1,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"

Audit and Monitoring Configuration#

1
plugins.security.audit.type: internal_opensearch
2
plugins.security.enable_snapshot_restore_privilege: true
3
plugins.security.check_snapshot_restore_write_privileges: true
4
plugins.security.restapi.roles_enabled:
5
  ["all_access", "security_rest_api_access"]

System Indices Configuration#

System indices are special indices used by OpenSearch plugins and features. Proper configuration ensures these indices are protected and managed correctly:

1
# System indices configuration
2
plugins.security.system_indices.enabled: true
3
plugins.security.system_indices.indices:
4
  [
5
    ".plugins-ml-*",
6
    ".opendistro-alerting-*",
7
    ".opendistro-anomaly-*",
8
    ".opendistro-reports-*",
9
    ".opensearch-notifications-*",
10
    ".opensearch-notebooks",
11
    ".opensearch-observability",
12
    ".ql-datasources",
13
    ".opendistro-asynchronous-search-*",
14
    ".replication-metadata-store",
15
    ".opensearch-knn-models",
16
    ".geospatial-ip2geo-data*",
17
    ".plugins-flow-framework-*",
18
  ]

Performance Optimization#

Memory Settings#

Proper memory configuration is crucial for performance:

1
# Memory optimization
2
bootstrap.memory_lock: true
3
indices.memory.index_buffer_size: 5%

Important: Set ES_HEAP_SIZE environment variable to 50% of available RAM (maximum 32GB).

Thread Pool Configuration#

Optimize thread pools for your workload:

1
# Thread pool optimization
2
thread_pool.write.queue_size: 200
3
thread_pool.search.queue_size: 500

Index Buffer Configuration#

Control memory usage for indexing operations:

1
# Index buffer settings
2
indices.memory.index_buffer_size: 5%
3
indices.memory.min_index_buffer_size: 48mb
4
indices.memory.max_index_buffer_size: 512mb

Cluster Architecture Diagram#

1
graph TB
2
    subgraph "OpenSearch Cluster"
3
        subgraph "Master Node"
4
            M[os1<br/>Master + Data<br/>172.17.14.79:9300]
5
        end
6

7
        subgraph "Data Nodes"
8
            D1[os2<br/>Data + Ingest<br/>172.17.14.89:9300]
9
            D2[os3<br/>Data + Ingest<br/>172.17.14.39:9300]
10
        end
11
    end
12

13
    subgraph "Client Access"
14
        HTTP[HTTP API<br/>Port 9200]
15
        TLS[TLS Encryption]
16
    end
17

18
    subgraph "Storage"
19
        DATA[Data Storage<br/>/var/lib/opensearch]
20
        LOGS[Log Storage<br/>/var/log/opensearch]
21
        BACKUP[Backup Repository<br/>/var/lib/opensearch/migration]
22
    end
23

24
    M -.-> D1
25
    M -.-> D2
26
    D1 -.-> D2
27

28
    HTTP --> M
29
    HTTP --> D1
30
    HTTP --> D2
31

32
    TLS --> HTTP
33

34
    M --> DATA
35
    D1 --> DATA
36
    D2 --> DATA
37

38
    M --> LOGS
39
    D1 --> LOGS
40
    D2 --> LOGS
41

42
    M --> BACKUP
43
    D1 --> BACKUP
44
    D2 --> BACKUP

Installation and Setup#

Prerequisites#

1
# System requirements
2
# - Java 11 or later
3
# - Minimum 4GB RAM per node
4
# - Sufficient disk space for data and logs
5

6
# Install OpenSearch
7
wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.x.x/opensearch-2.x.x-linux-x64.tar.gz
8
tar -xzf opensearch-2.x.x-linux-x64.tar.gz

Directory Setup#

1
# Create necessary directories
2
sudo mkdir -p /var/lib/opensearch
3
sudo mkdir -p /var/log/opensearch
4
sudo mkdir -p /var/lib/opensearch/migration
5
sudo mkdir -p /etc/opensearch/certs
6

7
# Set proper ownership
8
sudo chown -R opensearch:opensearch /var/lib/opensearch
9
sudo chown -R opensearch:opensearch /var/log/opensearch
10
sudo chown -R opensearch:opensearch /etc/opensearch

Certificate Setup#

Generate certificates for secure communication:

1
# Generate root CA
2
openssl genrsa -out root-ca-key.pem 2048
3
openssl req -new -x509 -sha256 -key root-ca-key.pem -out root-ca.pem -days 3650
4

5
# Generate node certificates
6
openssl genrsa -out node-key.pem 2048
7
openssl req -new -key node-key.pem -out node.csr
8
openssl x509 -req -in node.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node.pem -days 3650
9

10
# Generate admin certificate
11
openssl genrsa -out admin-key.pem 2048
12
openssl req -new -key admin-key.pem -out admin.csr
13
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 3650

Cluster Management#

Health Checks#

1
# Check cluster health
2
curl -k -u admin:password https://localhost:9200/_cluster/health?pretty
3

4
# Check node status
5
curl -k -u admin:password https://localhost:9200/_cat/nodes?v
6

7
# View cluster settings
8
curl -k -u admin:password https://localhost:9200/_cluster/settings?pretty

Common Management Tasks#

1
# List all indices
2
curl -k -u admin:password https://localhost:9200/_cat/indices?v
3

4
# Check cluster stats
5
curl -k -u admin:password https://localhost:9200/_cluster/stats?pretty
6

7
# Monitor thread pools
8
curl -k -u admin:password https://localhost:9200/_cat/thread_pool?v

Migration and Backup#

Snapshot Configuration#

The cluster is configured with a shared repository for backups:

1
path.repo: /var/lib/opensearch/migration

Creating Snapshots#

1
# Register snapshot repository
2
curl -k -u admin:password -X PUT "https://localhost:9200/_snapshot/backup_repo" -H 'Content-Type: application/json' -d'
3
{
4
  "type": "fs",
5
  "settings": {
6
    "location": "/var/lib/opensearch/migration"
7
  }
8
}'
9

10
# Create snapshot
11
curl -k -u admin:password -X PUT "https://localhost:9200/_snapshot/backup_repo/snapshot_1" -H 'Content-Type: application/json' -d'
12
{
13
  "indices": "*",
14
  "ignore_unavailable": true,
15
  "include_global_state": false
16
}'

Troubleshooting#

Common Issues#

Split Brain Prevention: Always use odd number of master-eligible nodes
Memory Issues: Ensure bootstrap.memory_lock: true and adequate heap size
Network Configuration: Verify firewall rules allow traffic on ports 9200 and 9300
Certificate Issues: Check certificate paths and permissions

Diagnostic Commands#

1
# Check if memory locking is working
2
curl -k -u admin:password https://localhost:9200/_nodes/stats/process?pretty
3

4
# Verify security plugin status
5
curl -k -u admin:password https://localhost:9200/_plugins/_security/authinfo?pretty
6

7
# Check cluster allocation
8
curl -k -u admin:password https://localhost:9200/_cluster/allocation/explain?pretty

Best Practices#

Production Recommendations#

Separate Master Nodes: Use dedicated master nodes in large clusters
Data Redundancy: Configure at least 1 replica for each index
Monitoring: Implement comprehensive monitoring and alerting
Backup Strategy: Regular automated snapshots
Security Hardening: Regular certificate rotation and access reviews

Performance Tuning#

JVM Settings: Optimize heap size and garbage collection
Index Settings: Configure appropriate refresh intervals
Shard Management: Balance primary and replica shards
Hardware: Use SSD storage for better I/O performance

This configuration provides a solid foundation for a production OpenSearch cluster with proper security, performance optimization, and high availability features.