Monitoring Root Command Execution with Auditd + Wazuh - Break It, Fix It, Learn It#

The Real Story: When Everything Breaks#

Today was messy. But you know what? That’s exactly how real learning happens.

I jumped back into my Wazuh lab after a break, tried to configure Auditd rules to monitor root-level command executions, and boom — 💀 Server crashed.

Why? A misconfigured rule sent the Wazuh manager into chaos.

But instead of panicking, I did what every Blue Teamer eventually learns to do:

🛠️ Rebuilt the broken environment
✅ Debugged and fixed rule conflicts
🔁 Reconnected agents
📈 Verified logs and alerts from Auditd

“Break it. Fix it. Learn. Repeat.” That’s the hacker way. That’s the SOC mindset.

Why This Matters#

In the real world:

🔹 Things break
🔹 Logs go missing
🔹 Rules misfire

But if you know why it broke, you’re already ahead.

Today wasn’t about perfect success. It was about resilience + technical clarity.

The Goal: Detect Root Command Execution#

We’re building a system to detect and alert on any command run by root (euid=0) using:

Auditd for syscall logging
Wazuh for alerting via custom rules

Environment Setup#

1
Environment:
2
  Agent: Kali Linux with Wazuh agent
3
  Manager: Wazuh OVA (Wazuh manager)
4
  Components:
5
    - Auditd installed on agent
6
    - Custom rules set on manager
7
    - Real-time alerting pipeline

Architecture Overview#

flowchart LR
    subgraph "Linux System (Kali)"
        A1[Root User] --> A2[System Calls]
        A2 --> A3[Kernel]
        A3 --> A4[Auditd]
        A4 --> A5[/var/log/audit/audit.log]
    end

    subgraph "Wazuh Agent"
        A5 --> B1[Log Collection]
        B1 --> B2[Agent Buffer]
        B2 --> B3[Secure Transport]
    end

    subgraph "Wazuh Manager"
        B3 --> C1[Log Reception]
        C1 --> C2[Decoder]
        C2 --> C3[Rules Engine]
        C3 --> C4[Alert Generation]
        C4 --> C5[Dashboard/API]
    end

    style A1 fill:#ff6b6b,stroke:#c92a2a
    style A4 fill:#4dabf7,stroke:#1864ab
    style C3 fill:#51cf66,stroke:#2b8a3e
    style C4 fill:#ffd43b,stroke:#fab005

Step-by-Step Implementation#

Step 1: Install Auditd on the Agent#

1
# Install Auditd and plugins
2
sudo apt install auditd audispd-plugins -y
3

4
# Enable and start the service
5
sudo systemctl enable auditd --now
6
sudo systemctl start auditd.service
7

8
# Verify installation
9
sudo systemctl status auditd

Step 2: Configure Auditd Rules#

First, check existing rules:

1
sudo auditctl -l

Now, add rules to track root commands:

1
# Edit the rules file
2
sudo nano /etc/audit/rules.d/audit.rules

Add these rules to monitor all commands executed with root privileges:

1
# Monitor all execve syscalls for root user (euid=0)
2
-a always,exit -F arch=b64 -F euid=0 -S execve -k root_commands
3
-a always,exit -F arch=b32 -F euid=0 -S execve -k root_commands
4

5
# Additional monitoring for specific sensitive commands
6
-a always,exit -F arch=b64 -F euid=0 -F exe=/usr/bin/passwd -k passwd_changes
7
-a always,exit -F arch=b64 -F euid=0 -F exe=/usr/sbin/useradd -k user_creation
8
-a always,exit -F arch=b64 -F euid=0 -F exe=/usr/sbin/userdel -k user_deletion
9
-a always,exit -F arch=b64 -F euid=0 -F exe=/bin/rm -F path=/etc/* -k config_deletion

Apply the rules:

1
# Load the new rules
2
sudo auditctl -R /etc/audit/rules.d/audit.rules
3

4
# Verify rules are loaded
5
sudo auditctl -l

Step 3: Configure Wazuh Manager#

Create the audit key list:

1
# Create directory if it doesn't exist
2
sudo mkdir -p /var/ossec/etc/lists
3

4
# Add our audit keys
5
echo "root_commands" | sudo tee /var/ossec/etc/lists/audit-keys
6
echo "passwd_changes" | sudo tee -a /var/ossec/etc/lists/audit-keys
7
echo "user_creation" | sudo tee -a /var/ossec/etc/lists/audit-keys
8
echo "user_deletion" | sudo tee -a /var/ossec/etc/lists/audit-keys
9
echo "config_deletion" | sudo tee -a /var/ossec/etc/lists/audit-keys

Step 4: Configure Log Collection#

Edit the Wazuh configuration:

1
sudo nano /var/ossec/etc/ossec.conf

Add the audit log monitoring configuration:

1
<ossec_config>
2
  <!-- Audit log monitoring -->
3
  <localfile>
4
    <location>/var/log/audit/audit.log</location>
5
    <log_format>audit</log_format>
6
  </localfile>
7
</ossec_config>

Step 5: Create Custom Detection Rules#

This is where things got interesting (and where my server crashed initially). Here’s the working configuration:

1
sudo nano /var/ossec/etc/rules/local_rules.xml

Add these rules:

1
<group name="audit,audit_command,">
2
  <!-- Base rule for audit commands -->
3
  <rule id="100001" level="3">
4
    <if_sid>80700</if_sid>
5
    <list field="audit.key" lookup="match_key_value" check_value="root_commands">etc/lists/audit-keys</list>
6
    <description>Audit: Command executed by root: $(audit.exe)</description>
7
    <group>audit_command,</group>
8
  </rule>
9

10
  <!-- High-priority root command execution -->
11
  <rule id="100002" level="7">
12
    <if_sid>100001</if_sid>
13
    <field name="audit.euid">^0$</field>
14
    <description>Audit: Root command execution detected - Command: $(audit.exe) - User: $(audit.auid)</description>
15
    <group>audit_command,pci_dss_10.2.2,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,tsc_CC6.1,tsc_CC7.2,tsc_CC7.3,</group>
16
  </rule>
17

18
  <!-- Critical: Password changes by root -->
19
  <rule id="100003" level="10">
20
    <if_sid>80700</if_sid>
21
    <list field="audit.key" lookup="match_key_value" check_value="passwd_changes">etc/lists/audit-keys</list>
22
    <description>Critical: Root user changed passwords - User affected: $(audit.auid)</description>
23
    <group>audit_command,authentication_failed,pci_dss_10.2.5,gdpr_IV_35.7,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.7,tsc_CC6.1,tsc_CC7.2,tsc_CC7.3,</group>
24
  </rule>
25

26
  <!-- User creation/deletion monitoring -->
27
  <rule id="100004" level="8">
28
    <if_sid>80700</if_sid>
29
    <list field="audit.key" lookup="match_key_value" check_value="user_creation">etc/lists/audit-keys</list>
30
    <description>Audit: New user created by root - Command: $(audit.exe)</description>
31
    <group>audit_command,account_changed,pci_dss_10.2.5,gdpr_IV_35.7,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.2,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
32
  </rule>
33

34
  <rule id="100005" level="9">
35
    <if_sid>80700</if_sid>
36
    <list field="audit.key" lookup="match_key_value" check_value="user_deletion">etc/lists/audit-keys</list>
37
    <description>Critical: User deleted by root - Command: $(audit.exe)</description>
38
    <group>audit_command,account_changed,pci_dss_10.2.5,gdpr_IV_35.7,gdpr_IV_32.2,hipaa_164.312.b,nist_800_53_AU.14,nist_800_53_AC.2,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
39
  </rule>
40

41
  <!-- Configuration file deletion -->
42
  <rule id="100006" level="10">
43
    <if_sid>80700</if_sid>
44
    <list field="audit.key" lookup="match_key_value" check_value="config_deletion">etc/lists/audit-keys</list>
45
    <description>Critical: Configuration file deleted by root - Path: $(audit.file.name)</description>
46
    <group>audit_command,config_changed,pci_dss_10.5.5,gdpr_IV_35.7,hipaa_164.312.b,nist_800_53_AU.14,tsc_CC6.1,tsc_CC7.2,tsc_CC7.3,</group>
47
  </rule>
48
</group>

Step 6: The Critical Restart (Where Things Can Break)#

1
# Test configuration first!
2
sudo /var/ossec/bin/ossec-analysisd -t
3

4
# If test passes, restart
5
sudo systemctl restart wazuh-manager
6

7
# Monitor logs for errors
8
sudo tail -f /var/ossec/logs/ossec.log

Troubleshooting: When Things Go Wrong#

Issue 1: Server Crash After Adding Rules#

Symptoms:

Wazuh manager becomes unresponsive
High CPU usage
Log flooding

Root Cause:

Incorrect rule syntax
Missing closing tags
Circular rule dependencies

Fix:

1
# Stop the service
2
sudo systemctl stop wazuh-manager
3

4
# Check rule syntax
5
sudo /var/ossec/bin/ossec-analysisd -t
6

7
# Fix syntax errors in local_rules.xml
8
# Remove problematic rules temporarily
9

10
# Start with minimal configuration
11
sudo systemctl start wazuh-manager

Issue 2: No Alerts Generated#

Symptoms:

Auditd logs show events
Wazuh doesn’t generate alerts

Debugging Steps:

1
# Check if agent is reading audit logs
2
sudo grep audit /var/ossec/logs/ossec.log
3

4
# Verify audit log format
5
sudo ausearch -k root_commands
6

7
# Test rule matching
8
sudo /var/ossec/bin/ossec-logtest
9
# Paste a sample audit log entry

Issue 3: Performance Impact#

Symptoms:

System slowdown
High disk I/O

Optimization:

1
# Adjust audit rules to be more specific
2
# Instead of monitoring all execve, target specific binaries
3

4
# Edit /etc/audit/rules.d/audit.rules
5
-a always,exit -F arch=b64 -F euid=0 -F exe=/bin/su -k root_commands
6
-a always,exit -F arch=b64 -F euid=0 -F exe=/usr/bin/sudo -k root_commands
7
-a always,exit -F arch=b64 -F euid=0 -F exe=/bin/bash -k root_commands

Testing the Detection#

Generate Test Events#

1
# Switch to root
2
sudo su
3

4
# Run various commands
5
whoami
6
ls /etc
7
cat /etc/passwd
8
useradd testuser
9
passwd testuser
10
userdel testuser

Verify Alert Generation#

Check alerts in real-time:

1
# On Wazuh Manager
2
sudo tail -f /var/ossec/logs/alerts/alerts.json | jq '.'

Example alert output:

1
{
2
  "timestamp": "2025-01-06T14:30:00.000+0000",
3
  "rule": {
4
    "level": 7,
5
    "description": "Audit: Root command execution detected - Command: /bin/ls - User: 1000",
6
    "id": "100002",
7
    "firedtimes": 1,
8
    "groups": ["audit", "audit_command", "pci_dss_10.2.2"]
9
  },
10
  "agent": {
11
    "id": "001",
12
    "name": "kali-agent"
13
  },
14
  "data": {
15
    "audit": {
16
      "euid": "0",
17
      "auid": "1000",
18
      "exe": "/bin/ls",
19
      "key": "root_commands"
20
    }
21
  }
22
}

Dashboard Integration#

Creating Custom Dashboards#

Access Wazuh Dashboard
- Navigate to Stack Management → Index Patterns
- Ensure wazuh-alerts-* exists

Create Visualization

1
Visualization Type: Data Table
2
Metrics: Count
3
Buckets:
4
  - Split Rows: rule.id
5
  - Split Rows: audit.exe
6
  - Split Rows: agent.name

Build Dashboard
- Add visualization for rule.id: 100002
- Create time-series chart for root command frequency
- Add top commands executed by root

Sample KQL Queries#

1
# All root commands in last 24h
2
rule.id: 100002 AND @timestamp >= now-24h
3

4
# Password changes by root
5
rule.id: 100003
6

7
# User management activities
8
rule.id: (100004 OR 100005)
9

10
# Critical configuration changes
11
rule.id: 100006 AND rule.level >= 9

Advanced Configuration#

1. Enhanced Audit Rules#

1
# Monitor specific system directories
2
-w /etc/shadow -p wa -k shadow_file_changes
3
-w /etc/passwd -p wa -k passwd_file_changes
4
-w /etc/sudoers -p wa -k sudoers_changes
5
-w /etc/ssh/sshd_config -p wa -k sshd_config_changes
6

7
# Monitor kernel modules
8
-w /sbin/insmod -p x -k kernel_modules
9
-w /sbin/rmmod -p x -k kernel_modules
10
-w /sbin/modprobe -p x -k kernel_modules
11

12
# Monitor system calls
13
-a always,exit -F arch=b64 -S open -F dir=/etc -F success=0 -k unauth_file_access

2. Correlation Rules#

1
<group name="audit_correlation,">
2
  <!-- Multiple failed root access attempts -->
3
  <rule id="100010" level="10" frequency="5" timeframe="120">
4
    <if_matched_sid>100002</if_matched_sid>
5
    <same_field>agent.id</same_field>
6
    <description>Multiple root commands executed in short time</description>
7
    <group>audit_command,attacks,</group>
8
  </rule>
9

10
  <!-- Suspicious pattern: user creation followed by password change -->
11
  <rule id="100011" level="12">
12
    <if_sid>100004</if_sid>
13
    <if_matched_sid>100003</if_matched_sid>
14
    <same_field>agent.id</same_field>
15
    <time_interval>60</time_interval>
16
    <description>Suspicious: User created and password immediately changed</description>
17
    <group>audit_command,attacks,</group>
18
  </rule>
19
</group>

3. Integration with Active Response#

1
<ossec_config>
2
  <active-response>
3
    <command>host-deny</command>
4
    <location>local</location>
5
    <level>12</level>
6
    <timeout>600</timeout>
7
  </active-response>
8
</ossec_config>

Performance Considerations#

System Impact#

Monitoring all root commands can impact system performance. Here’s how to optimize:

Selective Monitoring

1
# Instead of all execve calls, monitor specific high-risk commands
2
-a exit,always -F arch=b64 -F euid=0 -F exe=/usr/bin/curl -k network_commands
3
-a exit,always -F arch=b64 -F euid=0 -F exe=/usr/bin/wget -k network_commands

Rate Limiting

1
# Add to /etc/audit/auditd.conf
2
rate_limit = 500

Log Rotation

1
/var/log/audit/audit.log {
2
    daily
3
    rotate 7
4
    compress
5
    delaycompress
6
    postrotate
7
        /sbin/service auditd restart > /dev/null
8
    endscript
9
}

Security Best Practices#

Protect Audit Configuration
Terminal window
```
1
# Make audit rules immutable
2
-e 2
```

Monitor Audit Daemon

1
<rule id="100020" level="10">
2
  <decoded_as>syslog</decoded_as>
3
  <match>auditd.*stopped</match>
4
  <description>Critical: Audit daemon stopped</description>
5
</rule>

Regular Review
- Weekly review of root command patterns
- Monthly audit rule effectiveness assessment
- Quarterly false positive tuning

Compliance Mapping#

This configuration helps meet various compliance requirements:

Compliance	Requirement	How This Helps
PCI DSS 10.2.2	Monitor all actions by privileged users	Tracks all root commands
HIPAA 164.312(b)	Audit controls	Comprehensive audit logging
GDPR Article 32	Security of processing	Monitors system changes
NIST 800-53 AU-14	Session audit	Tracks privileged sessions
SOC 2 CC6.1	Logical access controls	Monitors privilege usage

Lessons Learned#

Test in isolation first - Always test new rules on a non-production system
Incremental changes - Add rules one at a time
Monitor performance - Watch system resources after changes
Document everything - Your future self will thank you
Have a rollback plan - Know how to quickly disable problematic rules

Conclusion#

Today’s journey from a crashed server to a working root command monitoring system taught me more than any perfect tutorial could. The real world is messy, systems break, and that’s where true learning happens.

Key takeaways:

🔧 Building resilience through breaking and fixing
📊 Understanding the full audit pipeline
🚨 Creating meaningful security alerts
🛡️ Protecting critical system operations

Remember: “Break it. Fix it. Learn. Repeat.”

That’s not just a motto—it’s the path to becoming a skilled security professional.

Next Steps#

Extend monitoring to specific application commands
Integrate with SOAR for automated response
Build ML models for anomaly detection
Create executive dashboards for compliance reporting

No shortcuts, no copy-paste fixes. Just raw, real learning. Building detection muscles one broken server at a time. 💪

Newsletter

Monitoring Root Command Execution with Auditd + Wazuh - Break It, Fix It, Learn It#

The Real Story: When Everything Breaks#

Why This Matters#

The Goal: Detect Root Command Execution#

Environment Setup#

Architecture Overview#

Step-by-Step Implementation#

Step 1: Install Auditd on the Agent#

Step 2: Configure Auditd Rules#

Step 3: Configure Wazuh Manager#

Step 4: Configure Log Collection#

Step 5: Create Custom Detection Rules#

Step 6: The Critical Restart (Where Things Can Break)#

Troubleshooting: When Things Go Wrong#

Issue 1: Server Crash After Adding Rules#

Issue 2: No Alerts Generated#

Issue 3: Performance Impact#

Testing the Detection#

Generate Test Events#

Verify Alert Generation#

Dashboard Integration#

Creating Custom Dashboards#

Sample KQL Queries#

Advanced Configuration#

1. Enhanced Audit Rules#

2. Correlation Rules#

3. Integration with Active Response#

Performance Considerations#

System Impact#

Security Best Practices#

Compliance Mapping#

Lessons Learned#

Conclusion#

Next Steps#