Comprehensive OpenSearch Deployment Guide#

This guide provides detailed instructions for deploying OpenSearch in both single-node and cluster configurations. OpenSearch is a powerful, open-source search and analytics engine that offers enterprise-grade security, powerful monitoring capabilities, and extensive customization options.

Introduction#

OpenSearch is a community-driven, open-source fork of Elasticsearch that provides a secure, high-performance search and analytics engine. Whether you’re deploying a single node for development or a multi-node cluster for production, this guide will help you set up and optimize your OpenSearch deployment.

Prerequisites#

System Requirements#

Operating System: Rocky Linux, RHEL, Ubuntu 20.04+, or Amazon Linux 2
RAM: Minimum 4GB per node (8GB recommended)
CPU: Minimum 2 cores per node
Storage: 20GB+ available space
Java: OpenJDK 11 or later

Required Packages#

1
# For RPM-based systems (Rocky Linux, RHEL)
2
sudo dnf install java-11-openjdk-devel unzip wget curl
3

4
# For Debian-based systems
5
sudo apt-get update
6
sudo apt-get install openjdk-11-jdk unzip wget curl

Network Requirements#

Open ports:
- 9200: HTTP API
- 9300: Node communication
- 9600: Performance Analyzer
- 5601: OpenSearch Dashboards (if installed)

Single Node Installation#

Add OpenSearch Repository#

1
# Create repository file
2
sudo bash -c 'cat > /etc/yum.repos.d/opensearch.repo << EOF
3
[opensearch]
4
name=OpenSearch
5
baseurl=https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/yum
6
enabled=1
7
gpgcheck=1
8
gpgkey=https://artifacts.opensearch.org/publickeys/opensearch.gpg
9
autorefresh=1
10
type=rpm-md
11
EOF'

Install OpenSearch#

1
sudo dnf install opensearch

Configure OpenSearch#

1
sudo bash -c 'cat > /etc/opensearch/opensearch.yml << EOF
2
cluster.name: opensearch-single
3
node.name: single-node
4
network.host: 0.0.0.0
5
discovery.type: single-node
6
path.data: /var/lib/opensearch
7
path.logs: /var/log/opensearch
8

9
# Memory Settings
10
bootstrap.memory_lock: true
11

12
# Security Settings
13
plugins.security.ssl.http.enabled: true
14
plugins.security.ssl.http.pemcert_filepath: certs/node.pem
15
plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem
16
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem
17
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem
18
plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem
19
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
20
plugins.security.ssl.transport.enforce_hostname_verification: false
21
plugins.security.allow_default_init_securityindex: true
22
plugins.security.authcz.admin_dn:
23
  - "CN=admin,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
24

25
# Performance Settings
26
indices.memory.index_buffer_size: 10%
27
thread_pool.write.queue_size: 200
28
thread_pool.search.queue_size: 500
29
EOF'

Generate Certificates#

1
# Create directory for certificates
2
sudo mkdir -p /etc/opensearch/certs
3
cd /etc/opensearch/certs
4

5
# Generate root CA
6
sudo openssl req -x509 -new -nodes -newkey rsa:2048 \
7
  -keyout root-ca.key -out root-ca.pem -days 3650 \
8
  -subj "/C=IN/L=Ahmedabad/O=Invinsense/OU=Invinsense/CN=Root CA"
9

10
# Generate node certificate
11
sudo openssl req -new -nodes -newkey rsa:2048 \
12
  -keyout node-key.pem -out node.csr \
13
  -subj "/C=IN/L=Ahmedabad/O=Invinsense/OU=Invinsense/CN=single-node"
14

15
sudo openssl x509 -req -in node.csr -CA root-ca.pem \
16
  -CAkey root-ca.key -CAcreateserial -out node.pem \
17
  -days 3650 -sha256
18

19
# Generate admin certificate
20
sudo openssl req -new -nodes -newkey rsa:2048 \
21
  -keyout admin-key.pem -out admin.csr \
22
  -subj "/C=IN/L=Ahmedabad/O=Invinsense/OU=Invinsense/CN=admin"
23

24
sudo openssl x509 -req -in admin.csr -CA root-ca.pem \
25
  -CAkey root-ca.key -CAcreateserial -out admin.pem \
26
  -days 3650 -sha256

Set Permissions#

1
sudo chown -R opensearch:opensearch /etc/opensearch/certs
2
sudo chmod 600 /etc/opensearch/certs/*.pem
3
sudo chmod 600 /etc/opensearch/certs/*.key

Configure System Settings#

1
# Set memory mapping limits
2
sudo bash -c 'cat >> /etc/sysctl.conf << EOF
3
vm.max_map_count=262144
4
vm.swappiness=1
5
EOF'
6

7
sudo sysctl -p
8

9
# Set memory lock limits
10
sudo bash -c 'cat > /etc/security/limits.d/opensearch.conf << EOF
11
opensearch soft memlock unlimited
12
opensearch hard memlock unlimited
13
opensearch soft nofile 65536
14
opensearch hard nofile 65536
15
opensearch soft nproc 4096
16
opensearch hard nproc 4096
17
EOF'

Start OpenSearch#

1
sudo systemctl enable opensearch
2
sudo systemctl start opensearch

Cluster Installation#

This section covers setting up a three-node cluster with one master node and two data nodes.

Master Node Configuration (os1)#

1
sudo bash -c 'cat > /etc/opensearch/opensearch.yml << EOF
2
# Basic cluster configuration
3
cluster.name: opensearch-cluster
4
node.name: os1
5
node.roles: [master, data]
6
network.host: 172.17.14.79
7
http.port: 9200
8
transport.port: 9300
9

10
# Discovery settings
11
discovery.seed_hosts: ["172.17.14.79:9300", "172.17.14.89:9300", "172.17.14.39:9300"]
12
cluster.initial_master_nodes: ["os1"]
13

14
# Memory and path settings
15
bootstrap.memory_lock: true
16
path.data: /var/lib/opensearch
17
path.logs: /var/log/opensearch
18

19
# Performance settings
20
indices.memory.index_buffer_size: 5%
21
thread_pool.write.queue_size: 200
22
thread_pool.search.queue_size: 500
23

24
# Security Configuration
25
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem
26
plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem
27
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
28
plugins.security.ssl.transport.enforce_hostname_verification: false
29
plugins.security.ssl.transport.resolve_hostname: false
30

31
plugins.security.ssl.http.enabled: true
32
plugins.security.ssl.http.pemcert_filepath: certs/node.pem
33
plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem
34
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem
35

36
plugins.security.allow_unsafe_democertificates: false
37
plugins.security.allow_default_init_securityindex: true
38

39
# Node Identity Management
40
plugins.security.nodes_dn:
41
  - "CN=opensearch-1,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
42
  - "CN=opensearch-2,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
43
  - "CN=opensearch-3,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
44

45
plugins.security.authcz.admin_dn:
46
  - "CN=admin,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
47

48
# System indices configuration
49
plugins.security.system_indices.enabled: true
50
plugins.security.system_indices.indices: [
51
  ".plugins-ml-*",
52
  ".opendistro-alerting-*",
53
  ".opendistro-anomaly-*",
54
  ".opendistro-reports-*",
55
  ".opensearch-notifications-*",
56
  ".opensearch-notebooks",
57
  ".opensearch-observability",
58
  ".ql-datasources",
59
  ".opendistro-asynchronous-search-*",
60
  ".replication-metadata-store",
61
  ".opensearch-knn-models",
62
  ".geospatial-ip2geo-data*",
63
  ".plugins-flow-framework-*"
64
]
65
EOF'

Data Node Configuration (os2 and os3)#

1
# Replace IP address and node name for each node
2
sudo bash -c 'cat > /etc/opensearch/opensearch.yml << EOF
3
# Basic cluster configuration
4
cluster.name: opensearch-cluster
5
node.name: os2  # Change to os3 for the third node
6
node.roles: [data, ingest]
7
network.host: 172.17.14.89  # Change to appropriate IP
8
http.port: 9200
9
transport.port: 9300
10

11
# Discovery settings
12
discovery.seed_hosts: ["172.17.14.79:9300", "172.17.14.89:9300", "172.17.14.39:9300"]
13
cluster.initial_master_nodes: ["os1"]
14

15
# Memory and path settings
16
bootstrap.memory_lock: true
17
path.data: /var/lib/opensearch
18
path.logs: /var/log/opensearch
19

20
# Performance settings
21
indices.memory.index_buffer_size: 5%
22
thread_pool.write.queue_size: 200
23
thread_pool.search.queue_size: 500
24

25
# Security Configuration
26
plugins.security.ssl.transport.pemcert_filepath: certs/node.pem
27
plugins.security.ssl.transport.pemkey_filepath: certs/node-key.pem
28
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
29
plugins.security.ssl.transport.enforce_hostname_verification: false
30
plugins.security.ssl.transport.resolve_hostname: false
31

32
plugins.security.ssl.http.enabled: true
33
plugins.security.ssl.http.pemcert_filepath: certs/node.pem
34
plugins.security.ssl.http.pemkey_filepath: certs/node-key.pem
35
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem
36

37
# Security Settings
38
plugins.security.allow_unsafe_democertificates: false
39
plugins.security.allow_default_init_securityindex: true
40

41
# Node Identity Management
42
plugins.security.nodes_dn:
43
  - "CN=opensearch-1,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
44
  - "CN=opensearch-2,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
45
  - "CN=opensearch-3,OU=Invinsense,O=Invinsense,L=Ahmedabad,C=IN"
46

47
# System indices configuration
48
plugins.security.system_indices.enabled: true
49
plugins.security.system_indices.indices: [
50
  ".plugins-ml-*",
51
  ".opendistro-alerting-*",
52
  ".opendistro-anomaly-*",
53
  ".opendistro-reports-*",
54
  ".opensearch-notifications-*",
55
  ".opensearch-notebooks",
56
  ".opensearch-observability",
57
  ".ql-datasources",
58
  ".opendistro-asynchronous-search-*",
59
  ".replication-metadata-store",
60
  ".opensearch-knn-models",
61
  ".geospatial-ip2geo-data*",
62
  ".plugins-flow-framework-*"
63
]
64
EOF'

Generate and Distribute Certificates#

Create a certificate generation script:

1
#!/bin/bash
2

3
CERT_DIR="/tmp/opensearch-certs"
4
mkdir -p "$CERT_DIR"
5

6
# Generate root CA
7
openssl req -x509 -new -nodes -newkey rsa:2048 \
8
  -keyout "$CERT_DIR/root-ca.key" -out "$CERT_DIR/root-ca.pem" \
9
  -days 3650 -subj "/C=IN/L=Ahmedabad/O=Invinsense/OU=Invinsense/CN=Root CA"
10

11
# Function to generate node certificate
12
generate_node_cert() {
13
    local node_name=$1
14
    openssl req -new -nodes -newkey rsa:2048 \
15
      -keyout "$CERT_DIR/$node_name-key.pem" -out "$CERT_DIR/$node_name.csr" \
16
      -subj "/C=IN/L=Ahmedabad/O=Invinsense/OU=Invinsense/CN=$node_name"
17

18
    openssl x509 -req -in "$CERT_DIR/$node_name.csr" \
19
      -CA "$CERT_DIR/root-ca.pem" -CAkey "$CERT_DIR/root-ca.key" \
20
      -CAcreateserial -out "$CERT_DIR/$node_name.pem" \
21
      -days 3650 -sha256
22
}
23

24
# Generate certificates for each node
25
generate_node_cert "opensearch-1"
26
generate_node_cert "opensearch-2"
27
generate_node_cert "opensearch-3"
28

29
# Generate admin certificate
30
openssl req -new -nodes -newkey rsa:2048 \
31
  -keyout "$CERT_DIR/admin-key.pem" -out "$CERT_DIR/admin.csr" \
32
  -subj "/C=IN/L=Ahmedabad/O=Invinsense/OU=Invinsense/CN=admin"
33

34
openssl x509 -req -in "$CERT_DIR/admin.csr" \
35
  -CA "$CERT_DIR/root-ca.pem" -CAkey "$CERT_DIR/root-ca.key" \
36
  -CAcreateserial -out "$CERT_DIR/admin.pem" \
37
  -days 3650 -sha256
38

39
# Set permissions
40
chmod 600 "$CERT_DIR"/*.pem "$CERT_DIR"/*.key

Distribute Certificates#

1
# On each node
2
sudo mkdir -p /etc/opensearch/certs
3
sudo cp /tmp/opensearch-certs/root-ca.pem /etc/opensearch/certs/
4
sudo cp /tmp/opensearch-certs/root-ca.key /etc/opensearch/certs/
5
sudo cp /tmp/opensearch-certs/admin*.pem /etc/opensearch/certs/
6

7
# On each node, copy the appropriate node certificate
8
# For os1
9
sudo cp /tmp/opensearch-certs/opensearch-1.pem /etc/opensearch/certs/node.pem
10
sudo cp /tmp/opensearch-certs/opensearch-1-key.pem /etc/opensearch/certs/node-key.pem
11

12
# For os2
13
sudo cp /tmp/opensearch-certs/opensearch-2.pem /etc/opensearch/certs/node.pem
14
sudo cp /tmp/opensearch-certs/opensearch-2-key.pem /etc/opensearch/certs/node-key.pem
15

16
# For os3
17
sudo cp /tmp/opensearch-certs/opensearch-3.pem /etc/opensearch/certs/node.pem
18
sudo cp /tmp/opensearch-certs/opensearch-3-key.pem /etc/opensearch/certs/node-key.pem
19

20
# Set permissions on all nodes
21
sudo chown -R opensearch:opensearch /etc/opensearch/certs
22
sudo chmod 750 /etc/opensearch/certs
23
sudo chmod 600 /etc/opensearch/certs/*

Configure System Settings on All Nodes#

1
# Configure sysctl settings
2
sudo bash -c 'cat >> /etc/sysctl.conf << EOF
3
vm.max_map_count=262144
4
vm.swappiness=1
5
EOF'
6

7
sudo sysctl -p
8

9
# Configure memory limits
10
sudo bash -c 'cat > /etc/security/limits.d/opensearch.conf << EOF
11
opensearch soft memlock unlimited
12
opensearch hard memlock unlimited
13
opensearch soft nofile 65536
14
opensearch hard nofile 65536
15
opensearch soft nproc 4096
16
opensearch hard nproc 4096
17
EOF'
18

19
# Create JVM options file
20
sudo mkdir -p /etc/opensearch/jvm.options.d/
21
sudo bash -c 'cat > /etc/opensearch/jvm.options.d/heap.options << EOF
22
-Xms2g
23
-Xmx2g
24
EOF'

Start the Cluster#

1
# On all nodes
2
sudo systemctl daemon-reload
3
sudo systemctl enable opensearch
4
sudo systemctl start opensearch
5

6
# Monitor logs
7
sudo tail -f /var/log/opensearch/opensearch-cluster.log

Security Configuration#

Create Security Configuration Directory#

1
sudo mkdir -p /etc/opensearch/opensearch-security

Configure Internal Users#

1
sudo bash -c 'cat > /etc/opensearch/opensearch-security/internal_users.yml << EOF
2
admin:
3
  hash: "$2y$12$XwI0HxUmgQpR5zIzWUKIBONRPvU1kRUFCaXUa2Z2iCGjBxbqpyKtG"  # Change this
4
  reserved: true
5
  backend_roles:
6
  - "admin"
7
  description: "Admin user"
8
kibanaserver:
9
  hash: "$2y$12$XwI0HxUmgQpR5zIzWUKIBONRPvU1kRUFCaXUa2Z2iCGjBxbqpyKtG"  # Change this
10
  reserved: true
11
  description: "OpenSearch Dashboards user"
12
EOF'

Important: Always change default password hashes in production environments.

Configure Roles#

1
sudo bash -c 'cat > /etc/opensearch/opensearch-security/roles.yml << EOF
2
admin:
3
  cluster:
4
    - UNLIMITED
5
  indices:
6
    "*":
7
      allowlist: ["*"]
8
      denylist: []
9
      authorized_indices: ["*"]
10
  tenants:
11
    admin_tenant: RW
12

13
kibanaserver:
14
  cluster:
15
    - CLUSTER_MONITOR
16
    - CLUSTER_COMPOSITE_OPS
17
  indices:
18
    "*":
19
      allowlist: [".kibana*", ".opensearch_dashboards*"]
20
      denylist: []
21
      authorized_indices: [".kibana*", ".opensearch_dashboards*"]
22
EOF'

Configure Role Mappings#

1
sudo bash -c 'cat > /etc/opensearch/opensearch-security/roles_mapping.yml << EOF
2
admin:
3
  users:
4
    - admin
5
  backend_roles:
6
    - admin
7
  hosts: []
8
  and_backend_roles: []
9

10
kibanaserver:
11
  users:
12
    - kibanaserver
13
  backend_roles: []
14
  hosts: []
15
  and_backend_roles: []
16
EOF'

Performance Tuning#

Memory Settings#

For nodes with 4GB RAM:

1
sudo bash -c 'cat > /etc/opensearch/jvm.options.d/heap.options << EOF
2
-Xms2g
3
-Xmx2g
4
EOF'

Thread Pool Settings#

Add the following to opensearch.yml:

1
thread_pool:
2
  write:
3
    queue_size: 200
4
  search:
5
    queue_size: 500

Field Data and Query Cache#

Add to opensearch.yml:

1
indices.fielddata.cache.size: 20%
2
indices.queries.cache.size: 10%

Disk I/O Settings#

Add to opensearch.yml:

1
index:
2
  merge:
3
    policy:
4
      floor_segment: 2mb
5
      max_merge_at_once: 5
6
      max_merged_segment: 5gb
7
  refresh_interval: 30s

Troubleshooting#

Memory Lock Issues#

1
# Check if memory lock is enabled
2
ulimit -l
3

4
# If unlimited is not shown, verify /etc/security/limits.conf:
5
sudo bash -c 'cat >> /etc/security/limits.conf << EOF
6
opensearch soft memlock unlimited
7
opensearch hard memlock unlimited
8
EOF'

Certificate Issues#

1
# Verify certificate permissions
2
ls -la /etc/opensearch/certs/
3

4
# Verify certificate content
5
openssl x509 -in /etc/opensearch/certs/node.pem -text -noout
6

7
# Check certificate chain
8
openssl verify -CAfile /etc/opensearch/certs/root-ca.pem /etc/opensearch/certs/node.pem

Cluster Formation Issues#

1
# Check cluster health
2
curl -k -X GET "https://localhost:9200/_cluster/health?pretty" -u admin:admin
3

4
# Check cluster state
5
curl -k -X GET "https://localhost:9200/_cluster/state?pretty" -u admin:admin
6

7
# Check node info
8
curl -k -X GET "https://localhost:9200/_nodes?pretty" -u admin:admin

Best Practices#

Security Best Practices#

Certificate Management:
- Rotate certificates every 12 months
- Use strong key sizes (minimum 2048 bits for RSA)
- Implement proper certificate revocation procedures
- Store private keys securely
Network Security:
- Use firewalls to restrict access to OpenSearch ports
- Implement VPC or network segmentation
- Use TLS 1.2 or higher for all communications
- Disable plaintext HTTP when possible
Authentication and Authorization:
- Use role-based access control (RBAC)
- Follow the principle of least privilege
- Regularly audit user access and permissions
- Implement strong password policies

Performance Best Practices#

Hardware Recommendations:
- Use SSDs for data nodes
- Provide adequate RAM (minimum 4GB per node)
- Use multiple CPUs/cores for better concurrent processing
- Ensure network bandwidth is sufficient
Indexing Best Practices:
- Use bulk indexing when possible
- Optimize refresh intervals based on workload
- Use appropriate shard sizes (10-50GB per shard)
- Implement proper mapping strategies
Search Optimization:
- Use filter context when possible
- Implement proper caching strategies
- Optimize query patterns
- Use scroll API for deep pagination

Monitoring Best Practices#

Key Metrics to Monitor:
- Cluster health status
- Node status and resource usage
- Index performance metrics
- Query performance metrics
- JVM heap usage
- Disk usage and I/O statistics
Logging:
- Implement log rotation
- Monitor error logs regularly
- Set appropriate log levels
- Archive logs for compliance and debugging
Alerting:
- Set up alerts for cluster state changes
- Monitor disk space usage
- Track JVM heap usage
- Monitor query latency
- Track failed node connections

Backup and Recovery#

Snapshot Repository Setup#

1
# Create snapshot repository
2
curl -X PUT "localhost:9200/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
3
{
4
  "type": "fs",
5
  "settings": {
6
    "location": "/mnt/backups"
7
  }
8
}'

Automated Backup Script#

1
#!/bin/bash
2
SNAPSHOT_NAME="snapshot_$(date +%Y%m%d_%H%M%S)"
3
curl -X PUT "localhost:9200/_snapshot/my_backup/${SNAPSHOT_NAME}?wait_for_completion=true"

Maintenance Procedures#

Rolling Restart Procedure#

1
# For each node in the cluster:
2
# 1. Disable shard allocation
3
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
4
{
5
  "persistent": {
6
    "cluster.routing.allocation.enable": "none"
7
  }
8
}'
9

10
# 2. Stop OpenSearch
11
sudo systemctl stop opensearch
12

13
# 3. Perform maintenance
14

15
# 4. Start OpenSearch
16
sudo systemctl start opensearch
17

18
# 5. Wait for node to join cluster
19
curl -X GET "localhost:9200/_cat/nodes"
20

21
# 6. Re-enable shard allocation
22
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
23
{
24
  "persistent": {
25
    "cluster.routing.allocation.enable": "all"
26
  }
27
}'

Index Lifecycle Management#

1
# Create index lifecycle policy
2
curl -X PUT "localhost:9200/_ilm/policy/my_policy" -H 'Content-Type: application/json' -d'
3
{
4
  "policy": {
5
    "phases": {
6
      "hot": {
7
        "actions": {
8
          "rollover": {
9
            "max_size": "50GB",
10
            "max_age": "30d"
11
          }
12
        }
13
      },
14
      "warm": {
15
        "min_age": "30d",
16
        "actions": {
17
          "shrink": {
18
            "number_of_shards": 1
19
          },
20
          "forcemerge": {
21
            "max_num_segments": 1
22
          }
23
        }
24
      },
25
      "cold": {
26
        "min_age": "60d",
27
        "actions": {
28
          "readonly": {}
29
        }
30
      },
31
      "delete": {
32
        "min_age": "90d",
33
        "actions": {
34
          "delete": {}
35
        }
36
      }
37
    }
38
  }
39
}'

Conclusion#

This comprehensive guide should help you deploy, secure, optimize, and maintain OpenSearch in production environments. Remember to regularly check the OpenSearch documentation for updates and new features that might affect your deployment strategy.

Production deployments require careful planning, monitoring, and maintenance. By following the practices outlined in this guide, you can build a robust, high-performance search infrastructure that meets your organization’s needs.