Practical Guide: Linux Kernel and Audit Log Monitoring with Wazuh#

Introduction#

Linux systems generate critical security information through kernel logs and audit events. This guide provides a practical, tested approach to collecting and monitoring these logs using Wazuh, based on real-world implementations and community experiences.

What We’ll Monitor#

Kernel Logs: Hardware events, system calls, security violations
Audit Events: User actions, file access, system changes
System Events: Authentication, network connections, process execution

Architecture Overview#

1
flowchart TB
2
    subgraph "Linux System"
3
        K[Linux Kernel] --> KL[Kernel Logs]
4
        A[Auditd] --> AL[Audit Logs]
5
        S[System Events] --> SL[System Logs]
6

7
        KL --> J[journald]
8
        AL --> AF[/var/log/audit/audit.log]
9
        SL --> SF[/var/log/syslog]
10
    end
11

12
    subgraph "Wazuh Agent"
13
        LC[Log Collector]
14
        J --> LC
15
        AF --> LC
16
        SF --> LC
17
        LC --> AB[Agent Buffer]
18
        AB --> T[TLS Transport]
19
    end
20

21
    subgraph "Wazuh Manager"
22
        T --> WM[Log Reception]
23
        WM --> D[Decoders]
24
        D --> R[Rules Engine]
25
        R --> AG[Alert Generation]
26
        AG --> FB[Filebeat]
27
    end
28

29
    subgraph "Wazuh Dashboard"
30
        FB --> WI[Wazuh Indexer]
31
        WI --> WD[Dashboard]
32
    end
33

34
    style K fill:#ff6b6b
35
    style A fill:#4dabf7
36
    style R fill:#51cf66
37
    style WD fill:#ffd43b

Part 1: Kernel Log Collection#

Method 1: Journald Integration (Recommended for Modern Systems)#

Basic Configuration#

Add to /var/ossec/etc/ossec.conf on the agent:

1
<ossec_config>
2
  <!-- Collect all journal logs including kernel -->
3
  <localfile>
4
    <location>journald</location>
5
    <log_format>journald</log_format>
6
  </localfile>
7
</ossec_config>

Advanced Kernel-Specific Configuration#

For kernel logs only:

1
<ossec_config>
2
  <!-- Kernel logs via journald -->
3
  <localfile>
4
    <location>journald</location>
5
    <log_format>journald</log_format>
6
    <filter field="_TRANSPORT">kernel</filter>
7
  </localfile>
8

9
  <!-- High-priority kernel events -->
10
  <localfile>
11
    <location>journald</location>
12
    <log_format>journald</log_format>
13
    <query>_TRANSPORT=kernel AND PRIORITY<=3</query>
14
    <target>kernel-critical</target>
15
  </localfile>
16
</ossec_config>

Testing Journald Filters#

Before applying, test your filters:

1
# View kernel messages
2
journalctl _TRANSPORT=kernel
3

4
# Shorthand for kernel messages
5
journalctl -k
6

7
# Real-time kernel monitoring
8
journalctl -k -f
9

10
# Check if wazuh user can access
11
sudo -u wazuh journalctl --since "1 minute ago" --no-pager

Method 2: Traditional Syslog Files#

For older systems or specific requirements:

1
<ossec_config>
2
  <!-- Kernel logs -->
3
  <localfile>
4
    <log_format>syslog</log_format>
5
    <location>/var/log/kern.log</location>
6
  </localfile>
7

8
  <!-- System messages (includes kernel) -->
9
  <localfile>
10
    <log_format>syslog</log_format>
11
    <location>/var/log/messages</location>
12
  </localfile>
13

14
  <!-- All syslog -->
15
  <localfile>
16
    <log_format>syslog</log_format>
17
    <location>/var/log/syslog</location>
18
  </localfile>
19
</ossec_config>

Troubleshooting Kernel Log Collection#

Issue: No Kernel Logs in Wazuh#

Check Permissions:

1
# Add wazuh to required groups
2
sudo usermod -a -G systemd-journal wazuh
3
sudo usermod -a -G adm wazuh
4

5
# Restart agent
6
sudo systemctl restart wazuh-agent

Verify Log Collection:

1
# Check agent log
2
tail -f /var/ossec/logs/ossec.log | grep -E "journald|kernel"
3

4
# Test log generation
5
echo "Test kernel message" | sudo tee /dev/kmsg
6

7
# Check if collected
8
grep "Test kernel message" /var/ossec/logs/archives/archives.log

Part 2: Audit Log Collection with Auditd#

Step 1: Install and Configure Auditd#

1
# Ubuntu/Debian
2
sudo apt-get install -y auditd audispd-plugins
3

4
# RHEL/CentOS
5
sudo yum install -y audit audit-libs
6

7
# Enable and start
8
sudo systemctl enable auditd
9
sudo systemctl start auditd

Step 2: Configure Wazuh Agent#

Add to /var/ossec/etc/ossec.conf:

1
<ossec_config>
2
  <localfile>
3
    <log_format>audit</log_format>
4
    <location>/var/log/audit/audit.log</location>
5
  </localfile>
6
</ossec_config>

Step 3: Create Audit Rules#

Monitor Root Command Execution#

Edit /etc/audit/rules.d/audit.rules:

1
# Monitor all commands by root (euid=0)
2
-a exit,always -F arch=b64 -F euid=0 -S execve -k audit-wazuh-c
3
-a exit,always -F arch=b32 -F euid=0 -S execve -k audit-wazuh-c
4

5
# Monitor specific user commands (uid=1000)
6
-a exit,always -F arch=b64 -F auid=1000 -S execve -k user-commands
7
-a exit,always -F arch=b32 -F auid=1000 -S execve -k user-commands
8

9
# File access monitoring
10
-w /etc/passwd -p wa -k passwd-changes
11
-w /etc/shadow -p wa -k shadow-changes
12
-w /etc/sudoers -p wa -k sudoers-changes
13

14
# System calls
15
-a always,exit -F arch=b64 -S open,openat -F exit=-EPERM -k access-denied
16
-a always,exit -F arch=b64 -S unlink,rename -k file-deletion
17

18
# Kernel module monitoring
19
-w /sbin/insmod -p x -k kernel-modules
20
-w /sbin/rmmod -p x -k kernel-modules
21
-w /sbin/modprobe -p x -k kernel-modules

Apply rules:

1
# Load rules
2
sudo auditctl -R /etc/audit/rules.d/audit.rules
3

4
# Verify rules
5
sudo auditctl -l

Step 4: Configure Wazuh Server#

Update Audit Keys List#

The Wazuh server maintains a list of audit keys at /var/ossec/etc/lists/audit-keys:

1
# View current keys
2
cat /var/ossec/etc/lists/audit-keys
3

4
# Add custom keys
5
echo "user-commands:command" | sudo tee -a /var/ossec/etc/lists/audit-keys
6
echo "passwd-changes:write" | sudo tee -a /var/ossec/etc/lists/audit-keys
7
echo "kernel-modules:command" | sudo tee -a /var/ossec/etc/lists/audit-keys
8

9
# Restart manager
10
sudo systemctl restart wazuh-manager

Create Custom Rules#

Add to /var/ossec/etc/rules/local_rules.xml:

1
<group name="audit,">
2
  <!-- Enhanced root command detection -->
3
  <rule id="100002" level="6">
4
    <if_sid>80792</if_sid>
5
    <field name="audit.euid">0</field>
6
    <description>Root command execution: $(audit.exe) by user $(audit.auid)</description>
7
    <group>audit_command,</group>
8
  </rule>
9

10
  <!-- Detect suspicious commands -->
11
  <rule id="100003" level="8">
12
    <if_sid>100002</if_sid>
13
    <field name="audit.exe">nc|ncat|netcat|tcpdump|wireshark</field>
14
    <description>Suspicious command by root: $(audit.exe)</description>
15
    <group>audit_command,suspicious_command,</group>
16
  </rule>
17

18
  <!-- File modification alerts -->
19
  <rule id="100004" level="7">
20
    <if_sid>80784</if_sid>
21
    <field name="audit.key">passwd-changes</field>
22
    <description>Password file modified: $(audit.file.name)</description>
23
    <group>audit_watch_write,</group>
24
  </rule>
25

26
  <!-- Kernel module changes -->
27
  <rule id="100005" level="9">
28
    <if_sid>80792</if_sid>
29
    <field name="audit.key">kernel-modules</field>
30
    <description>Kernel module operation: $(audit.exe)</description>
31
    <group>audit_command,kernel_modules,</group>
32
  </rule>
33
</group>

Part 3: Real-World Testing#

Generate Test Events#

1
# Test root commands
2
sudo su -
3
whoami
4
ls /etc/shadow
5
cat /etc/passwd
6

7
# Test kernel module operations
8
sudo modprobe -v scsi_debug
9
sudo rmmod scsi_debug
10

11
# Test file modifications
12
sudo touch /etc/test-file
13
sudo rm /etc/test-file
14

15
# Generate kernel messages
16
echo "Security test message" | sudo tee /dev/kmsg

Verify Alerts#

1
# Check alerts on manager
2
sudo tail -f /var/ossec/logs/alerts/alerts.json | jq '.'
3

4
# Filter specific alerts
5
sudo grep "100002\|100003\|100004\|100005" /var/ossec/logs/alerts/alerts.log

Part 4: Dashboard Configuration#

Create Visualizations#

Security Events View:
- Navigate to Modules > Security Events
- Add filters:
  - rule.groups: audit
  - rule.level >= 6
Audit Dashboard Queries:

1
# Root commands
2
rule.id: 100002
3

4
# Suspicious commands
5
rule.id: 100003
6

7
# File modifications
8
rule.groups: audit_watch_write
9

10
# Kernel events
11
data.audit.key: kernel-modules
12

13
# Failed access attempts
14
data.audit.success: no

Custom Dashboard Panels:
- Top users executing root commands
- Command frequency timeline
- File modification heat map
- Failed access attempts by source

Part 5: Common Issues and Solutions#

Issue 1: No Audit Events in Dashboard#

Solution:

1
# Check if auditd is running
2
sudo systemctl status auditd
3

4
# Verify log file exists and growing
5
ls -la /var/log/audit/audit.log
6
tail -f /var/log/audit/audit.log
7

8
# Check agent configuration
9
grep -A5 "audit" /var/ossec/etc/ossec.conf
10

11
# Restart agent
12
sudo systemctl restart wazuh-agent

Issue 2: High Log Volume#

Solution:

1
<!-- Rate limit in ossec.conf -->
2
<ossec_config>
3
  <localfile>
4
    <log_format>audit</log_format>
5
    <location>/var/log/audit/audit.log</location>
6
    <frequency>10</frequency>
7
  </localfile>
8
</ossec_config>

Issue 3: Missing Specific Events#

Debug Process:

1
# Test audit rules
2
sudo auditctl -m "Test audit message"
3

4
# Search audit log
5
sudo ausearch -m USER_CMD -ts recent
6

7
# Check key-based search
8
sudo ausearch -k audit-wazuh-c
9

10
# Use ossec-logtest
11
echo 'type=SYSCALL msg=audit(1234567890.123:456): arch=c000003e syscall=59 success=yes exit=0 a0=1234 a1=5678 items=2 ppid=1000 pid=2000 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/usr/bin/ls" key="audit-wazuh-c"' | sudo /var/ossec/bin/ossec-logtest

Part 6: Performance Optimization#

Audit Rule Optimization#

1
# Limit rules to specific users
2
-a exit,always -F arch=b64 -F auid>=1000 -F auid<2000 -S execve -k user-commands
3

4
# Exclude noisy processes
5
-a exit,never -F exe=/usr/bin/bash
6
-a exit,always -F arch=b64 -F euid=0 -S execve -k root-commands
7

8
# Use syscall filtering
9
-a exit,always -F arch=b64 -S socket -F a0=2 -k network-connections

Wazuh Agent Tuning#

1
<ossec_config>
2
  <!-- Buffer configuration -->
3
  <client_buffer>
4
    <disabled>no</disabled>
5
    <queue_size>5000</queue_size>
6
    <events_per_second>500</events_per_second>
7
  </client_buffer>
8
</ossec_config>

Part 7: Security Use Cases#

1. Privilege Escalation Detection#

1
<rule id="100010" level="10">
2
  <if_sid>100002</if_sid>
3
  <field name="audit.auid" negate="yes">0</field>
4
  <field name="audit.exe">su|sudo|pkexec</field>
5
  <description>Privilege escalation: $(audit.auid) to root via $(audit.exe)</description>
6
  <group>audit_command,privilege_escalation,</group>
7
</rule>

2. Unauthorized Access Attempts#

1
<rule id="100011" level="8">
2
  <if_sid>80784</if_sid>
3
  <field name="audit.success">no</field>
4
  <description>Failed file access attempt: $(audit.file.name)</description>
5
  <group>audit_watch_read,access_denied,</group>
6
</rule>

3. Suspicious Network Tools#

1
<rule id="100012" level="9">
2
  <if_sid>80792</if_sid>
3
  <field name="audit.exe">nmap|masscan|zmap|nikto</field>
4
  <description>Network scanning tool detected: $(audit.exe)</description>
5
  <group>audit_command,recon,</group>
6
</rule>

Best Practices#

Start Small: Begin with basic rules and expand gradually
Test Thoroughly: Verify rules in test environment first
Monitor Performance: Watch system load with audit enabled
Document Rules: Maintain documentation for audit rules
Regular Review: Periodically review and update rules

Conclusion#

This practical guide provides a working foundation for Linux kernel and audit log monitoring with Wazuh. By implementing these configurations, you’ll gain deep visibility into system activities, enabling effective threat detection and compliance monitoring.

Remember: Security monitoring is an iterative process. Start with the basics, test thoroughly, and expand based on your specific security requirements.

Additional Resources#

Building comprehensive security monitoring, one log at a time. Stay vigilant! 🛡️