Complete Data Prepper 2.11.0 Setup Guide - Wazuh to OpenSearch

Table of contents#

Overview#

Complete setup guide for migrating from Logstash to Data Prepper 2.11.0 for Wazuh security event ingestion into OpenSearch.

Prerequisites#

OpenSearch cluster running on https://localhost:9200
Wazuh manager generating alerts at /var/ossec/logs/alerts/alerts.json
Root access to the system

1. Install Data Prepper 2.11.0#

1
# Download and install Data Prepper 2.11.0
2
cd /tmp
3
wget https://github.com/opensearch-project/data-prepper/releases/download/2.11.0/opensearch-data-prepper-2.11.0-linux-x64.tar.gz
4

5
# Extract to /opt
6
sudo tar -xzf opensearch-data-prepper-2.11.0-linux-x64.tar.gz -C /opt/
7
sudo mv /opt/opensearch-data-prepper-2.11.0 /opt/data-prepper
8

9
# Set ownership
10
sudo chown -R root:root /opt/data-prepper

2. Create Data Prepper User#

1
# Create dedicated user for Data Prepper
2
sudo useradd -r -s /bin/bash -d /var/lib/dataprepper -m dataprepper
3

4
# Create necessary directories
5
sudo mkdir -p /etc/data-prepper
6
sudo mkdir -p /var/log/data-prepper
7
sudo mkdir -p /var/lib/dataprepper
8

9
# Set ownership
10
sudo chown -R dataprepper:dataprepper /etc/data-prepper
11
sudo chown -R dataprepper:dataprepper /var/log/data-prepper
12
sudo chown -R dataprepper:dataprepper /var/lib/dataprepper

3. Create Data Prepper Configuration#

Main Configuration File#

1
sudo tee /etc/data-prepper/data-prepper-config.yaml > /dev/null <<EOF
2
ssl: false
3
keyStoreFilePath: ""
4
keyStorePassword: ""
5
privateKeyPassword: ""
6
serverPort: 4900
7
metricRegistries:
8
  - Prometheus
9
metricTags:
10
  service: "data-prepper"
11
EOF

Wazuh Index Template#

1
curl -o /etc/data-prepper/wazuh.json https://packages.wazuh.com/integrations/opensearch/4.x-2.x/dashboards/wz-os-4.x-2.x-template.json

4. Create Working Pipeline Configuration#

1
sudo tee /etc/data-prepper/pipelines-wazuh.yaml > /dev/null <<'EOF'
2
wazuh-alerts-pipeline:
3
  source:
4
    file:
5
      path: "/var/ossec/logs/alerts/alerts.json"
6
      # Use no codec (plain text) and parse JSON in processor
7
      record_type: "event"
8

9
  processor:
10
    # Parse each line as JSON (equivalent to Logstash codec behavior)
11
    - parse_json:
12
        source: "message"
13

14
  sink:
15
    - opensearch:
16
        hosts:
17
          - "https://localhost:9200"
18
        # Update these credentials for your environment
19
        username: "admin"
20
        password: "YOUR_OPENSEARCH_PASSWORD"
21
        index: "wazuh-alerts-4.x-%{yyyy.MM.dd}"
22
        insecure: true
23
        template_file: "/etc/data-prepper/wazuh.json"
24
        template_type: "index-template"
25
        bulk_size: 1000
26
        flush_timeout: 5000
27
EOF

5. Set Proper Permissions#

1
# Set ownership of all configuration files
2
sudo chown -R dataprepper:dataprepper /etc/data-prepper
3

4
# Make Data Prepper executable accessible
5
sudo chmod +x /opt/data-prepper/bin/data-prepper
6

7
# Ensure dataprepper user can read Wazuh alerts
8
sudo usermod -a -G ossec dataprepper
9
# OR if ossec group doesn't exist:
10
sudo chmod 644 /var/ossec/logs/alerts/alerts.json

6. Create Systemd Service#

1
sudo tee /etc/systemd/system/data-prepper.service > /dev/null <<EOF
2
[Unit]
3
Description=OpenSearch Data Prepper
4
After=network.target
5

6
[Service]
7
Type=simple
8
User=dataprepper
9
Group=dataprepper
10
ExecStart=/opt/data-prepper/bin/data-prepper /etc/data-prepper/pipelines-wazuh.yaml /etc/data-prepper/data-prepper-config.yaml
11
Restart=always
12
RestartSec=10
13
StandardOutput=journal
14
StandardError=journal
15
SyslogIdentifier=data-prepper
16

17
# Java heap settings
18
Environment="JAVA_OPTS=-Xms1g -Xmx2g"
19

20
[Install]
21
WantedBy=multi-user.target
22
EOF
23

24
# Reload systemd and enable service
25
sudo systemctl daemon-reload
26
sudo systemctl enable data-prepper

7. Test Configuration#

Manual Test#

1
# Test the configuration manually
2
sudo -u dataprepper bash -c '
3
  echo "Testing Data Prepper with Wazuh pipeline..."
4
  /opt/data-prepper/bin/data-prepper /etc/data-prepper/pipelines-wazuh.yaml /etc/data-prepper/data-prepper-config.yaml
5
'

Expected Success Indicators#

No ClassCastException errors
“Initialized OpenSearch sink” message
“Starting file source with /var/ossec/logs/alerts/alerts.json path” message
No authentication errors

Verify Data Ingestion#

1
# Check if indices are created
2
curl -k -u "admin:YOUR_PASSWORD" "https://localhost:9200/_cat/indices/wazuh-alerts-4.x-*?v"
3

4
# Check document count
5
curl -k -u "admin:YOUR_PASSWORD" "https://localhost:9200/wazuh-alerts-4.x-*/_count" | jq .
6

7
# View sample documents
8
curl -k -u "admin:YOUR_PASSWORD" "https://localhost:9200/wazuh-alerts-4.x-*/_search?size=1&pretty"

8. Start Data Prepper Service#

1
# Start the service
2
sudo systemctl start data-prepper
3

4
# Check status
5
sudo systemctl status data-prepper
6

7
# View logs
8
sudo journalctl -u data-prepper -f

9. Production Security Hardening#

Environment Variables for Credentials#

1
# Create environment file
2
sudo tee /etc/data-prepper/environment > /dev/null <<EOF
3
OPENSEARCH_USERNAME=admin
4
OPENSEARCH_PASSWORD=your_secure_password_here
5
EOF
6

7
sudo chown dataprepper:dataprepper /etc/data-prepper/environment
8
sudo chmod 600 /etc/data-prepper/environment

Update Pipeline for Environment Variables#

1
sudo tee /etc/data-prepper/pipelines-wazuh-secure.yaml > /dev/null <<'EOF'
2
wazuh-alerts-pipeline:
3
  source:
4
    file:
5
      path: "/var/ossec/logs/alerts/alerts.json"
6
      record_type: "event"
7

8
  processor:
9
    - parse_json:
10
        source: "message"
11

12
  sink:
13
    - opensearch:
14
        hosts:
15
          - "https://localhost:9200"
16
        username: "${OPENSEARCH_USERNAME}"
17
        password: "${OPENSEARCH_PASSWORD}"
18
        index: "wazuh-alerts-4.x-%{yyyy.MM.dd}"
19
        insecure: false  # Set to true only for testing
20
        cert: "/path/to/opensearch/certs/root-ca.pem"
21
        template_file: "/etc/data-prepper/wazuh.json"
22
        template_type: "index-template"
23
        bulk_size: 1000
24
        flush_timeout: 5000
25
EOF

Update Systemd Service for Environment Variables#

1
sudo systemctl edit data-prepper --full
2
# Add this line under [Service]:
3
# EnvironmentFile=/etc/data-prepper/environment

10. Monitoring and Troubleshooting#

Key Log Locations#

Data Prepper logs: sudo journalctl -u data-prepper -f
Configuration directory: /etc/data-prepper/
Data directory: /var/lib/dataprepper/

Common Issues#

Authentication errors: Check OpenSearch credentials
File permission errors: Ensure dataprepper user can read Wazuh logs
JSON parsing errors: Verify Wazuh alert format matches expected structure
Memory issues: Adjust JAVA_OPTS in systemd service

Performance Tuning#

1
# In pipeline configuration, adjust:
2
bulk_size: 1000 # Increase for higher throughput
3
flush_timeout: 5000 # Decrease for lower latency
4
request_timeout: "30s" # Increase for large batches

Equivalent Logstash vs Data Prepper#

Logstash Configuration#

1
input {
2
  file {
3
    codec => "json"
4
    path => "/var/ossec/logs/alerts/alerts.json"
5
    mode => "tail"
6
  }
7
}
8
output {
9
  opensearch {
10
    hosts => ["https://localhost:9200"]
11
    username => "admin"
12
    password => "password"
13
    index => "wazuh-alerts-4.x-%{+YYYY.MM.dd}"
14
  }
15
}

Data Prepper Equivalent#

1
wazuh-alerts-pipeline:
2
  source:
3
    file:
4
      path: "/var/ossec/logs/alerts/alerts.json"
5
      record_type: "event"
6
  processor:
7
    - parse_json:
8
        source: "message"
9
  sink:
10
    - opensearch:
11
        hosts: ["https://localhost:9200"]
12
        username: "admin"
13
        password: "password"
14
        index: "wazuh-alerts-4.x-%{yyyy.MM.dd}"

Migration Checklist#

✅ Migration Complete: Your Logstash → Data Prepper migration for XDR security event ingestion is now operational!

Import Wazuh Dashboards#

Follow these steps to import the Wazuh dashboards for OpenSearch.

Run the command below to download the Wazuh dashboard file for OpenSearch:

If you are accessing the OpenSearch dashboard from a Linux or macOS system:

1
wget https://packages.wazuh.com/integrations/opensearch/4.x-2.x/dashboards/wz-os-4.x-2.x-dashboards.ndjson

If you are accessing the Opensearch dashboard from a Windows system (run the command using Powershell):

1
Invoke-WebRequest -Uri "https://packages.wazuh.com/integrations/opensearch/4.x-2.x/dashboards/wz-os-4.x-2.x-dashboards.ndjson" -OutFile "allDashboards.ndjson"

In OpenSearch Dashboards, navigate to Management > Dashboards management.
Click on Saved Objects and click Import.
Click on the Import icon, browse your files, and select the dashboard file.
Click the Import button to start importing then click Done.
To find the imported dashboards, navigate to Dashboard under OpenSearch Dashboards.