Deploying OpenSearch with Podman and Quadlet for Containerized Search at Scale#

OpenSearch is a powerful open-source search and analytics engine that many organizations use for log analysis, full-text search, and observability. While Kubernetes is commonly used for deploying OpenSearch clusters, there are simpler alternatives for smaller deployments or environments where Kubernetes is not available. This guide demonstrates how to deploy OpenSearch using Podman and Quadlet for a lightweight, secure, and systemd-integrated solution.

Architecture Overview#

This deployment uses a layered architecture that provides stability, security, and integration with the host system:

1
graph TD
2
    User[User Application] -->|Search Requests| OS[OpenSearch Container]
3
    Admin[Administrator] -->|Management| SD[Systemd Service]
4
    SD -->|Controls| PC[Podman Container]
5
    PC -->|Runs| OS
6
    OS -->|Stores| Data[Persistent Volume]
7

8
    subgraph "Host System"
9
        SD
10
        PC
11
        Data
12
    end
13

14
    subgraph "Container"
15
        OS
16
    end

The architecture consists of:

Podman Container: Root-less container runtime without daemon
Quadlet: Configuration standard for systemd container integration
OpenSearch: Search and analytics engine
Persistent Storage: Volume mapping for durable data
Systemd Integration: For service management and automatic restarts

Prerequisites#

Before starting the implementation, ensure you have:

Linux Distribution: Fedora, RHEL, CentOS Stream, or other Linux with Podman 4.0+
Hardware Requirements:
- Minimum 2 cores and 4GB RAM (8GB+ recommended)
- 20GB available storage
Software:
- Podman 4.0 or newer
- Systemd 248 or newer (for Quadlet support)
- User with sudo access

Implementation Steps#

Our implementation follows a systematic approach for deploying OpenSearch with Podman and Quadlet.

1. Installing and Configuring Podman#

First, let’s make sure Podman is properly installed and configured:

1
# Install Podman if not already installed
2
sudo dnf install -y podman podman-docker
3

4
# Check Podman version (should be 4.0+)
5
podman --version
6

7
# Enable and start the Podman socket
8
systemctl --user enable --now podman.socket
9

10
# Verify socket status
11
systemctl --user status podman.socket

2. Creating a Persistent Volume for OpenSearch Data#

OpenSearch requires persistent storage to maintain data across container restarts:

1
# Create a directory for persistent data
2
mkdir -p ~/.local/share/opensearch-data
3

4
# Set appropriate permissions
5
chmod 777 ~/.local/share/opensearch-data

3. Testing OpenSearch with Podman#

Before creating the Quadlet file, let’s test running OpenSearch with Podman directly:

1
# Run OpenSearch as a single-node cluster
2
podman run -d \
3
  --name opensearch-node1 \
4
  -e cluster.name=opensearch-cluster \
5
  -e node.name=opensearch-node1 \
6
  -e discovery.type=single-node \
7
  -e "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" \
8
  -e "plugins.security.disabled=true" \
9
  -e bootstrap.memory_lock=false \
10
  -e "path.data=/usr/share/opensearch/data" \
11
  --ulimit nofile=65536:65536 \
12
  -v ~/.local/share/opensearch-data:/usr/share/opensearch/data:Z,rw \
13
  -p 19200:9200 \
14
  -p 19600:9600 \
15
  --user "$(id -u):$(id -g)" \
16
  docker.io/opensearchproject/opensearch:latest
17

18
# Check if the container is running
19
podman ps
20

21
# View logs to verify startup
22
podman logs -f opensearch-node1

4. Creating a Quadlet Configuration for OpenSearch#

Now, let’s create a Quadlet file to define our OpenSearch container configuration for systemd:

1
# Create Quadlet directory if it doesn't exist
2
mkdir -p ~/.config/containers/systemd
3

4
# Create the Quadlet configuration file
5
cat << EOF > ~/.config/containers/systemd/opensearch.container
6
[Unit]
7
Description=OpenSearch Container
8
After=network-online.target
9
Wants=network-online.target
10

11
[Container]
12
Image=docker.io/opensearchproject/opensearch:latest
13
Network=host
14
PublishPort=19200:9200
15
PublishPort=19600:9600
16
Volume=~/.local/share/opensearch-data:/usr/share/opensearch/data:Z
17
Environment=cluster.name=opensearch-cluster
18
Environment=node.name=opensearch-node1
19
Environment=discovery.type=single-node
20
Environment=OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
21
Environment=plugins.security.disabled=true
22
Environment=path.data=/usr/share/opensearch/data
23
Environment=bootstrap.memory_lock=false
24
Ulimit=nofile=65536:65536
25
User=$(id -u):$(id -g)
26

27
[Service]
28
Restart=always
29
TimeoutStartSec=900
30
TimeoutStopSec=70
31

32
[Install]
33
WantedBy=default.target
34
EOF
35

36
# Reload systemd to detect the new Quadlet file
37
systemctl --user daemon-reload

5. Managing OpenSearch with Systemd#

With the Quadlet file in place, you can now manage OpenSearch using systemd:

1
# Enable the service to start at login
2
systemctl --user enable container-opensearch.service
3

4
# Enable lingering to ensure services start at boot
5
loginctl enable-linger $USER
6

7
# Start the OpenSearch service
8
systemctl --user start container-opensearch.service
9

10
# Check service status
11
systemctl --user status container-opensearch.service

6. Testing OpenSearch Functionality#

Once OpenSearch is running, let’s verify that it’s working properly:

1
# Test the OpenSearch REST API
2
curl -X GET "http://localhost:19200/"
3

4
# Create a test index
5
curl -X PUT "http://localhost:19200/test-index"
6

7
# Add a document to the index
8
curl -X POST "http://localhost:19200/test-index/_doc" \
9
  -H 'Content-Type: application/json' \
10
  -d '{"title": "Test Document", "content": "This is a test document for OpenSearch."}'
11

12
# Search for documents
13
curl -X GET "http://localhost:19200/test-index/_search?q=test"

7. Securing OpenSearch#

For production use, you should enable security features in OpenSearch:

1
# Stop the existing container
2
systemctl --user stop container-opensearch.service
3

4
# Update Quadlet configuration with security settings
5
cat << EOF > ~/.config/containers/systemd/opensearch.container
6
[Unit]
7
Description=OpenSearch Container
8
After=network-online.target
9
Wants=network-online.target
10

11
[Container]
12
Image=docker.io/opensearchproject/opensearch:latest
13
Network=host
14
PublishPort=19200:9200
15
PublishPort=19600:9600
16
Volume=~/.local/share/opensearch-data:/usr/share/opensearch/data:Z
17
Environment=cluster.name=opensearch-cluster
18
Environment=node.name=opensearch-node1
19
Environment=discovery.type=single-node
20
Environment=OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
21
Environment=plugins.security.disabled=false
22
Environment=OPENSEARCH_INITIAL_ADMIN_PASSWORD=StrongPasswordHere
23
Environment=path.data=/usr/share/opensearch/data
24
Environment=bootstrap.memory_lock=false
25
Ulimit=nofile=65536:65536
26
User=$(id -u):$(id -g)
27

28
[Service]
29
Restart=always
30
TimeoutStartSec=900
31
TimeoutStopSec=70
32

33
[Install]
34
WantedBy=default.target
35
EOF
36

37
# Reload and restart the service
38
systemctl --user daemon-reload
39
systemctl --user restart container-opensearch.service
40

41
# Test secured API with basic auth
42
curl -u admin:StrongPasswordHere -X GET "http://localhost:19200/"

Advanced Configuration#

Multi-node Cluster Configuration#

For a more robust deployment, you can set up a multi-node cluster:

1
# Create Quadlet file for node 1
2
cat << EOF > ~/.config/containers/systemd/opensearch-1.container
3
[Unit]
4
Description=OpenSearch Node 1
5
After=network-online.target
6
Wants=network-online.target
7

8
[Container]
9
Image=docker.io/opensearchproject/opensearch:latest
10
Network=host
11
PublishPort=19201:9200
12
PublishPort=19601:9600
13
Volume=~/.local/share/opensearch-data-1:/usr/share/opensearch/data:Z
14
Environment=cluster.name=opensearch-cluster
15
Environment=node.name=opensearch-node1
16
Environment=discovery.seed_hosts=localhost:19301,localhost:19302
17
Environment=cluster.initial_master_nodes=opensearch-node1,opensearch-node2
18
Environment=OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
19
Environment=path.data=/usr/share/opensearch/data
20
Environment=network.host=0.0.0.0
21
Environment=transport.port=19301
22
Environment=http.port=9200
23
Ulimit=nofile=65536:65536
24
User=$(id -u):$(id -g)
25

26
[Service]
27
Restart=always
28

29
[Install]
30
WantedBy=default.target
31
EOF
32

33
# Create Quadlet file for node 2
34
cat << EOF > ~/.config/containers/systemd/opensearch-2.container
35
[Unit]
36
Description=OpenSearch Node 2
37
After=network-online.target
38
Wants=network-online.target
39

40
[Container]
41
Image=docker.io/opensearchproject/opensearch:latest
42
Network=host
43
PublishPort=19202:9200
44
PublishPort=19602:9600
45
Volume=~/.local/share/opensearch-data-2:/usr/share/opensearch/data:Z
46
Environment=cluster.name=opensearch-cluster
47
Environment=node.name=opensearch-node2
48
Environment=discovery.seed_hosts=localhost:19301,localhost:19302
49
Environment=cluster.initial_master_nodes=opensearch-node1,opensearch-node2
50
Environment=OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m
51
Environment=path.data=/usr/share/opensearch/data
52
Environment=network.host=0.0.0.0
53
Environment=transport.port=19302
54
Environment=http.port=9200
55
Ulimit=nofile=65536:65536
56
User=$(id -u):$(id -g)
57

58
[Service]
59
Restart=always
60

61
[Install]
62
WantedBy=default.target
63
EOF

Adding OpenSearch Dashboards#

To add OpenSearch Dashboards for visualization:

1
# Create Quadlet file for OpenSearch Dashboards
2
cat << EOF > ~/.config/containers/systemd/opensearch-dashboards.container
3
[Unit]
4
Description=OpenSearch Dashboards
5
After=container-opensearch.service
6
Requires=container-opensearch.service
7

8
[Container]
9
Image=docker.io/opensearchproject/opensearch-dashboards:latest
10
PublishPort=15601:5601
11
Environment=OPENSEARCH_HOSTS=["http://localhost:19200"]
12
Environment=DISABLE_SECURITY_DASHBOARDS_PLUGIN=true
13

14
[Service]
15
Restart=always
16

17
[Install]
18
WantedBy=default.target
19
EOF

Monitoring and Management#

Checking Container Logs#

You can check the logs of your OpenSearch container using:

1
# View recent logs
2
systemctl --user status container-opensearch.service
3

4
# Stream logs
5
journalctl --user -f -u container-opensearch.service
6

7
# Or use podman logs
8
podman logs -f opensearch

Container Resource Management#

You can adjust resource limits in your Quadlet file:

1
# Update resource limits in ~/.config/containers/systemd/opensearch.container
2
[Container]
3
# ...existing configuration...
4
Environment=OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g
5
CPUQuota=200%
6
MemoryLimit=2G

Troubleshooting#

Common Issues and Solutions#

Permission Denied Errors#

If you encounter permission issues with the data directory:

1
# Fix permissions
2
chmod 777 ~/.local/share/opensearch-data
3
# Or use a more secure approach with appropriate ownership
4
chown -R $(id -u):$(id -g) ~/.local/share/opensearch-data
5
chmod 700 ~/.local/share/opensearch-data

OpenSearch Fails to Start#

If OpenSearch doesn’t start properly:

1
# Check for virtual memory limits
2
cat /proc/sys/vm/max_map_count
3
# If less than 262144, set it higher:
4
sudo sysctl -w vm.max_map_count=262144
5
# Make persistent by adding to /etc/sysctl.conf
6
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf

Service Does Not Start Automatically#

If the service doesn’t start at boot:

1
# Verify lingering is enabled
2
loginctl show-user $USER | grep Linger
3

4
# Enable if needed
5
loginctl enable-linger $USER
6

7
# Verify systemd service is enabled
8
systemctl --user is-enabled container-opensearch.service

Security Best Practices#

Network Security#

Use host firewall to restrict access to OpenSearch ports
Configure TLS for transport and HTTP layers
Place OpenSearch behind a reverse proxy for additional security

Authentication and Authorization#

Create custom users with limited permissions
Use role-based access control
Regularly rotate admin passwords

Auditing and Monitoring#

Enable audit logging
Monitor for unauthorized access attempts
Set up alerts for security events

Conclusion#

Deploying OpenSearch with Podman and Quadlet provides a lightweight, secure, and systemd-integrated search solution without the complexity of Docker or Kubernetes. This approach offers several advantages:

Daemonless Operation: No background daemon required, reducing resource usage
Rootless Security: Run containers as regular users, enhancing security
Systemd Integration: Native service management with dependencies and logging
Portability: Works across various Linux distributions with minimal dependencies
Simplified Management: Clear configuration files with Quadlet

Whether you’re setting up a small development environment or a production system with modest requirements, this deployment method offers a robust solution for running OpenSearch in containers.