OpenSearch NFS Mount Configuration Guide#

This guide provides detailed instructions for configuring NFS mounts for OpenSearch data migration across cluster nodes. Properly configured NFS mounts enable seamless data transfer while ensuring consistent permissions and ownership across your OpenSearch/Wazuh indexer infrastructure.

Prerequisites#

Before you begin, ensure you have:

OpenSearch nodes with compatible UID/GID for the opensearch user
NFS server running and accessible at the specified IP
nfs-utils installed on all nodes

Installation#

If NFS utilities are not already installed on your systems:

1
# On RHEL/CentOS/Fedora
2
sudo yum -y install nfs-utils
3

4
# On Debian/Ubuntu
5
sudo apt -y install nfs-common

Steps for Each OpenSearch Node#

Follow these steps on each node in your OpenSearch cluster:

1. Stop OpenSearch Service (if running)#

1
sudo systemctl stop opensearch

2. Sync UID/GID Across Nodes#

Consistency in user and group IDs is crucial for shared NFS access. Ensure all nodes use the same UID/GID for the opensearch user:

1
# Check current IDs
2
id opensearch
3

4
# If needed, modify UID/GID (for example, to match 996:993)
5
sudo usermod -u 996 opensearch
6
sudo groupmod -g 993 opensearch
7

8
# Fix ownership of OpenSearch directories
9
sudo chown -R opensearch:opensearch /var/lib/opensearch/
10
sudo chown -R opensearch:opensearch /etc/opensearch/
11
sudo chown -R opensearch:opensearch /var/log/opensearch/

3. Prepare Mount Directory#

Create and configure the mount point with appropriate permissions:

1
# Unmount if already mounted
2
sudo umount /var/lib/opensearch/migration 2>/dev/null || true
3

4
# Create mount directory with proper permissions
5
sudo mkdir -p /var/lib/opensearch/migration
6
sudo chown opensearch:opensearch /var/lib/opensearch/migration
7
sudo chmod 755 /var/lib/opensearch/migration

Connect to the NFS server:

1
# Mount the NFS share
2
sudo mount -t nfs 172.17.14.126:/var/lib/wazuh-indexer/migration /var/lib/opensearch/migration
3

4
# Verify mount and permissions
5
ls -la /var/lib/opensearch/migration

5. Configure Persistent Mount#

Add an entry to /etc/fstab for automatic mounting on system reboot:

1
echo "172.17.14.126:/var/lib/wazuh-indexer/migration /var/lib/opensearch/migration nfs defaults,_netdev,soft,timeo=30 0 0" | sudo tee -a /etc/fstab

Note: The soft and timeo options prevent system hangs if the NFS server becomes unavailable, while _netdev ensures the filesystem is mounted after the network is online.

6. Verify Write Access#

Test that the OpenSearch user can write to the mounted directory:

1
sudo -u opensearch touch /var/lib/opensearch/migration/test_$(hostname)
2
ls -la /var/lib/opensearch/migration/test_$(hostname)

7. Update OpenSearch Configuration#

If necessary, add the migration path to the OpenSearch configuration:

1
sudo nano /etc/opensearch/opensearch.yml

You may need to add path configurations like:

1
# Migration path config
2
path.repo: ["/var/lib/opensearch/migration"]

8. Restart OpenSearch#

Once configuration is complete, restart the OpenSearch service:

1
sudo systemctl start opensearch
2
sudo systemctl status opensearch

Troubleshooting#

Permission Issues#

If you encounter permission-related problems:

1
# Verify UID/GID consistency across nodes
2
id opensearch
3

4
# Check numeric UIDs of files on the mount
5
ls -ln /var/lib/opensearch/migration
6

7
# Temporarily set more permissive permissions for testing
8
sudo chmod 777 /var/lib/opensearch/migration

Mount Issues#

If the NFS mount fails:

1
# Check if NFS server is reachable
2
ping 172.17.14.126
3

4
# Verify the NFS share is exported on the server
5
showmount -e 172.17.14.126
6

7
# Check system logs
8
dmesg | grep nfs
9
sudo tail -f /var/log/messages

OpenSearch Issues#

If OpenSearch fails to start after configuration:

1
# Check logs for errors
2
sudo tail -f /var/log/opensearch/opensearch-cluster.log

Common issues include:

Incorrect permissions on the NFS mount
Path not correctly specified in opensearch.yml
Network connectivity problems between nodes and NFS server

Security Considerations#

When implementing NFS mounts for OpenSearch, keep these security best practices in mind:

The NFS mount should only be accessible within a secure network
Consider using NFSv4 with Kerberos authentication for production environments
Restrict mount permissions to only what’s necessary (755 or more restrictive)
Always verify correct ownership and permissions before starting OpenSearch
Use firewall rules to restrict NFS traffic to specific hosts

Performance Optimization#

For optimal NFS performance with OpenSearch:

Mount Options: Consider these additional mount options for better performance:
```
1
rsize=1048576,wsize=1048576,hard,noatime
```
Network Configuration: Use a dedicated network interface for NFS traffic if possible
NFS Server Tuning: On the NFS server, increase the number of NFS daemon threads:
Terminal window
```
1
echo "options nfs threads=16" | sudo tee /etc/modprobe.d/nfs.conf
```

Conclusion#

Properly configured NFS mounts enable efficient data migration and sharing between OpenSearch nodes. By ensuring consistent user/group permissions and following security best practices, you can maintain data integrity while leveraging the flexibility of networked storage for your OpenSearch clusters.

For more information on OpenSearch configuration and administration, refer to the official OpenSearch documentation.