Complete Guide to Linux Kernel Log Monitoring with Wazuh: From Zero to Hero#

Introduction: Why Kernel Logs Matter#

Linux kernel logs are the heartbeat of your system’s security posture. They reveal:

🔍 Hardware failures and driver issues
🛡️ Security violations and exploit attempts
📊 System performance degradation
🚨 Memory corruption and segmentation faults
🔐 Unauthorized kernel module loading

Today, we’ll build a comprehensive kernel log monitoring system using Wazuh that catches everything from rootkit installations to memory exploits.

Architecture Overview#

flowchart TB
    subgraph "Linux Kernel Space"
        K1[Kernel Ring Buffer]
        K2[printk Messages]
        K3[Security Events]
        K4[Driver Messages]
        K5[System Calls]
    end
    
    subgraph "User Space Logging"
        U1[/dev/kmsg]
        U2[klogd/rsyslog]
        U3[systemd-journald]
        U4[/var/log/kern.log]
        U5[dmesg Buffer]
    end
    
    subgraph "Wazuh Agent"
        W1[Log Collector]
        W2[journald Module]
        W3[syslog Module]
        W4[Command Module]
        W5[Agent Buffer]
    end
    
    subgraph "Wazuh Manager"
        M1[Log Analysis]
        M2[Decoders]
        M3[Rules Engine]
        M4[Alert Generation]
        M5[Integration]
    end
    
    K1 --> U1
    K2 --> U1
    K3 --> U1
    K4 --> U1
    K5 --> U1
    
    U1 --> U2
    U1 --> U3
    U2 --> U4
    U1 --> U5
    
    U3 --> W2
    U4 --> W3
    U5 --> W4
    
    W2 --> W1
    W3 --> W1
    W4 --> W1
    W1 --> W5
    
    W5 --> M1
    M1 --> M2
    M2 --> M3
    M3 --> M4
    M4 --> M5
    
    style K1 fill:#ff6b6b,stroke:#c92a2a
    style K3 fill:#ff6b6b,stroke:#c92a2a
    style W1 fill:#4dabf7,stroke:#1864ab
    style M3 fill:#51cf66,stroke:#2b8a3e
    style M4 fill:#ffd43b,stroke:#fab005

Part 1: Modern Approach - Journald Integration#

Understanding Journald for Kernel Logs#

Modern Linux distributions use systemd-journald to collect all system logs, including kernel messages. This provides structured logging with metadata.

Basic Journald Configuration#

Edit /var/ossec/etc/ossec.conf:

1
<ossec_config>
2
  <!-- Basic kernel log collection via journald -->
3
  <localfile>
4
    <location>journald</location>
5
    <log_format>journald</log_format>
6
  </localfile>
7
</ossec_config>

Advanced Journald Filtering#

For production environments, filter kernel logs to reduce noise:

1
<ossec_config>
2
  <!-- Only kernel transport messages -->
3
  <localfile>
4
    <location>journald</location>
5
    <log_format>journald</log_format>
6
    <filter field="_TRANSPORT">kernel</filter>
7
  </localfile>
8

9
  <!-- High-priority kernel messages only -->
10
  <localfile>
11
    <location>journald</location>
12
    <log_format>journald</log_format>
13
    <query>_TRANSPORT=kernel AND PRIORITY<=3</query>
14
    <target>critical-kernel</target>
15
  </localfile>
16

17
  <!-- Security-related kernel events -->
18
  <localfile>
19
    <location>journald</location>
20
    <log_format>journald</log_format>
21
    <query>_TRANSPORT=kernel AND (MESSAGE~".*SELinux.*" OR MESSAGE~".*audit.*" OR MESSAGE~".*security.*")</query>
22
    <target>kernel-security</target>
23
  </localfile>
24
</ossec_config>

Manual Testing of Journald Filters#

Before applying configurations, test your filters:

1
# View all kernel messages
2
journalctl _TRANSPORT=kernel
3

4
# Test priority filter
5
journalctl _TRANSPORT=kernel PRIORITY=0..3
6

7
# Security-related kernel events
8
journalctl _TRANSPORT=kernel | grep -E "(SELinux|audit|security)"
9

10
# Real-time kernel log monitoring
11
journalctl -f _TRANSPORT=kernel
12

13
# Last 100 kernel messages with timestamps
14
journalctl -k -n 100 --no-pager

Part 2: Traditional Syslog Approach#

Classic Kernel Log Files#

For systems using traditional syslog:

1
<ossec_config>
2
  <!-- Ubuntu/Debian kernel logs -->
3
  <localfile>
4
    <log_format>syslog</log_format>
5
    <location>/var/log/kern.log</location>
6
  </localfile>
7

8
  <!-- RHEL/CentOS system messages -->
9
  <localfile>
10
    <log_format>syslog</log_format>
11
    <location>/var/log/messages</location>
12
  </localfile>
13

14
  <!-- Generic syslog -->
15
  <localfile>
16
    <log_format>syslog</log_format>
17
    <location>/var/log/syslog</location>
18
  </localfile>
19
</ossec_config>

Multi-file Monitoring#

Monitor multiple log locations for comprehensive coverage:

1
<ossec_config>
2
  <!-- Wildcard pattern for all system logs -->
3
  <localfile>
4
    <location>/var/log/*.log</location>
5
    <log_format>syslog</log_format>
6
  </localfile>
7

8
  <!-- Rotated kernel logs -->
9
  <localfile>
10
    <location>/var/log/kern.log*</location>
11
    <log_format>syslog</log_format>
12
    <exclude>/var/log/kern.log*.gz</exclude>
13
  </localfile>
14
</ossec_config>

Part 3: Advanced Kernel Monitoring Techniques#

Direct Kernel Ring Buffer Access#

1
<ossec_config>
2
  <!-- Real-time dmesg monitoring -->
3
  <localfile>
4
    <log_format>command</log_format>
5
    <command>dmesg -T --follow --level=err,crit,alert,emerg</command>
6
    <alias>kernel-errors</alias>
7
  </localfile>
8

9
  <!-- Periodic full kernel buffer snapshot -->
10
  <localfile>
11
    <log_format>command</log_format>
12
    <command>dmesg -T | tail -100</command>
13
    <frequency>300</frequency>
14
    <alias>kernel-snapshot</alias>
15
  </localfile>
16
</ossec_config>

Kernel Module Monitoring#

Track kernel module loading/unloading:

1
<ossec_config>
2
  <!-- Monitor loaded modules -->
3
  <localfile>
4
    <log_format>command</log_format>
5
    <command>lsmod | md5sum</command>
6
    <frequency>600</frequency>
7
    <alias>loaded-modules-hash</alias>
8
  </localfile>
9

10
  <!-- Detailed module info -->
11
  <localfile>
12
    <log_format>full_command</log_format>
13
    <command>for mod in $(lsmod | awk '{print $1}' | tail -n +2); do modinfo $mod 2>/dev/null | grep -E "^(filename|description|author|license):" | sed "s/^/$mod: /"; done</command>
14
    <frequency>3600</frequency>
15
    <alias>module-details</alias>
16
  </localfile>
17
</ossec_config>

Security-Focused Kernel Monitoring#

1
<ossec_config>
2
  <!-- OOM killer events -->
3
  <localfile>
4
    <log_format>command</log_format>
5
    <command>dmesg | grep -i "killed process" | tail -10</command>
6
    <frequency>60</frequency>
7
    <alias>oom-killer</alias>
8
  </localfile>
9

10
  <!-- Segmentation faults -->
11
  <localfile>
12
    <log_format>command</log_format>
13
    <command>dmesg | grep -i segfault | tail -10</command>
14
    <frequency>60</frequency>
15
    <alias>segfaults</alias>
16
  </localfile>
17

18
  <!-- USB device connections -->
19
  <localfile>
20
    <log_format>command</log_format>
21
    <command>dmesg | grep -E "USB.*: (new|Product|Manufacturer)" | tail -20</command>
22
    <frequency>30</frequency>
23
    <alias>usb-devices</alias>
24
  </localfile>
25
</ossec_config>

Part 4: Custom Rules for Kernel Events#

Creating Kernel-Specific Rules#

Add to /var/ossec/etc/rules/local_rules.xml:

1
<group name="kernel,linux,">
2
  <!-- Kernel module loading -->
3
  <rule id="110001" level="7">
4
    <decoded_as>syslog</decoded_as>
5
    <match>kernel: module</match>
6
    <regex>loaded|inserted</regex>
7
    <description>Kernel module loaded: $(log)</description>
8
    <group>kernel_modules,</group>
9
  </rule>
10

11
  <!-- Critical: Unknown module -->
12
  <rule id="110002" level="10">
13
    <if_sid>110001</if_sid>
14
    <regex>TAINT</regex>
15
    <description>Tainted kernel module loaded - possible rootkit</description>
16
    <group>kernel_modules,rootkit,</group>
17
  </rule>
18

19
  <!-- Memory corruption -->
20
  <rule id="110003" level="12">
21
    <decoded_as>syslog</decoded_as>
22
    <match>kernel:</match>
23
    <regex>segfault|general protection fault</regex>
24
    <description>Kernel detected memory corruption: $(log)</description>
25
    <group>kernel_security,exploit,</group>
26
  </rule>
27

28
  <!-- OOM Killer -->
29
  <rule id="110004" level="9">
30
    <decoded_as>syslog</decoded_as>
31
    <match>kernel: Out of memory: Kill process</match>
32
    <description>Out of Memory: Process killed by kernel</description>
33
    <group>kernel_resource,system_error,</group>
34
  </rule>
35

36
  <!-- USB device detection -->
37
  <rule id="110005" level="5">
38
    <decoded_as>syslog</decoded_as>
39
    <match>kernel: usb</match>
40
    <regex>new USB device found</regex>
41
    <description>New USB device connected</description>
42
    <group>kernel_hardware,</group>
43
  </rule>
44

45
  <!-- Suspicious USB -->
46
  <rule id="110006" level="10">
47
    <if_sid>110005</if_sid>
48
    <regex>BadUSB|Rubber Ducky|WHID</regex>
49
    <description>Suspicious USB device detected</description>
50
    <group>kernel_hardware,attack,</group>
51
  </rule>
52

53
  <!-- SELinux violations -->
54
  <rule id="110007" level="8">
55
    <decoded_as>syslog</decoded_as>
56
    <match>kernel: audit</match>
57
    <regex>denied|violation</regex>
58
    <description>SELinux security violation</description>
59
    <group>kernel_security,access_denied,</group>
60
  </rule>
61

62
  <!-- Kernel panic indicators -->
63
  <rule id="110008" level="12">
64
    <decoded_as>syslog</decoded_as>
65
    <match>kernel:</match>
66
    <regex>panic|Oops|BUG:|Unable to handle kernel</regex>
67
    <description>Kernel panic or critical error detected</description>
68
    <group>kernel_error,system_error,</group>
69
  </rule>
70
</group>

Advanced Correlation Rules#

1
<group name="kernel_correlation,">
2
  <!-- Multiple module loads in short time -->
3
  <rule id="110020" level="12" frequency="5" timeframe="60">
4
    <if_matched_sid>110001</if_matched_sid>
5
    <same_field>agent.id</same_field>
6
    <description>Multiple kernel modules loaded rapidly - possible rootkit installation</description>
7
    <group>kernel_modules,rootkit,</group>
8
  </rule>
9

10
  <!-- Repeated segfaults -->
11
  <rule id="110021" level="14" frequency="3" timeframe="300">
12
    <if_matched_sid>110003</if_matched_sid>
13
    <same_field>agent.id</same_field>
14
    <description>Multiple memory corruption events - active exploitation attempt</description>
15
    <group>kernel_security,active_attack,</group>
16
  </rule>
17
</group>

Part 5: Performance Optimization#

Agent Buffer Configuration#

Optimize for high-volume kernel logs:

1
<ossec_config>
2
  <client_buffer>
3
    <disabled>no</disabled>
4
    <queue_size>20000</queue_size>
5
    <events_per_second>2000</events_per_second>
6
  </client_buffer>
7

8
  <!-- Specific buffer for kernel logs -->
9
  <localfile>
10
    <location>journald</location>
11
    <log_format>journald</log_format>
12
    <filter field="_TRANSPORT">kernel</filter>
13
    <out_format>json</out_format>
14
    <target>kernel</target>
15
    <reconnect_time>10</reconnect_time>
16
  </localfile>
17
</ossec_config>

Rate Limiting and Filtering#

1
<ossec_config>
2
  <!-- Priority-based filtering -->
3
  <localfile>
4
    <location>journald</location>
5
    <log_format>journald</log_format>
6
    <query>_TRANSPORT=kernel AND (PRIORITY=0 OR PRIORITY=1 OR PRIORITY=2)</query>
7
    <target>kernel-critical</target>
8
    <only-future-events>yes</only-future-events>
9
  </localfile>
10

11
  <!-- Exclude noisy drivers -->
12
  <localfile>
13
    <location>/var/log/kern.log</location>
14
    <log_format>syslog</log_format>
15
    <exclude>nvidia|nouveau|iwlwifi</exclude>
16
  </localfile>
17
</ossec_config>

Part 6: Troubleshooting Guide#

Common Issues and Solutions#

Issue 1: No Kernel Logs Received#

Diagnosis:

1
# Check if wazuh user can access logs
2
sudo -u wazuh journalctl --since "1 minute ago" --no-pager
3

4
# Verify permissions
5
ls -la /var/log/kern.log
6
groups wazuh

Solution:

1
# Add wazuh to systemd-journal group
2
sudo usermod -a -G systemd-journal wazuh
3
sudo usermod -a -G adm wazuh
4

5
# Restart agent
6
sudo systemctl restart wazuh-agent

Issue 2: High CPU Usage#

Diagnosis:

1
# Check agent resource usage
2
top -p $(pgrep -f wazuh-agent)
3

4
# Review log collection rate
5
sudo /var/ossec/bin/agent_control -i $(hostname) -s

Solution:

1
<!-- Implement rate limiting -->
2
<localfile>
3
  <location>journald</location>
4
  <log_format>journald</log_format>
5
  <filter field="_TRANSPORT">kernel</filter>
6
  <frequency>10</frequency>
7
  <max-log-size>1MB</max-log-size>
8
</localfile>

Issue 3: Missing Critical Events#

Debug Process:

1
# Test log collection
2
echo "Test kernel message" | sudo tee /dev/kmsg
3

4
# Check if received
5
sudo tail -f /var/ossec/logs/archives/archives.log | grep "Test kernel"
6

7
# Verify decoder
8
echo 'Jan  6 10:00:00 hostname kernel: [12345.678] Test message' | sudo /var/ossec/bin/ossec-logtest

Part 7: Real-World Security Scenarios#

Scenario 1: Rootkit Detection#

1
# Simulate suspicious module load
2
sudo insmod /path/to/suspicious.ko
3

4
# Expected alert
5
Rule: 110002 (level 10) -> 'Tainted kernel module loaded - possible rootkit'

Scenario 2: Memory Exploitation#

1
# Generate segfault
2
./vulnerable_program
3

4
# Expected alert
5
Rule: 110003 (level 12) -> 'Kernel detected memory corruption'

Scenario 3: Resource Exhaustion#

1
# Trigger OOM killer
2
stress-ng --vm 1 --vm-bytes 100G --timeout 10s
3

4
# Expected alert
5
Rule: 110004 (level 9) -> 'Out of Memory: Process killed by kernel'

Part 8: Integration with SIEM Dashboard#

Creating Kernel Security Dashboard#

Key Visualizations#

Kernel Events Timeline

1
Query: rule.groups: kernel
2
Visualization: Line chart
3
X-axis: @timestamp
4
Y-axis: Count

Top Kernel Security Events

1
Query: rule.groups: kernel AND rule.level >= 7
2
Visualization: Data table
3
Metrics: Count
4
Buckets: rule.description

Module Loading Activity

1
Query: rule.id: 110001
2
Visualization: Bar chart
3
Split series: agent.name

Alert Queries#

1
# Critical kernel security events
2
rule.groups: kernel AND rule.level >= 10
3

4
# Memory corruption attempts
5
rule.id: 110003 OR rule.id: 110021
6

7
# Suspicious USB devices
8
rule.id: 110006
9

10
# Kernel module changes
11
rule.groups: kernel_modules
12

13
# System stability issues
14
rule.groups: kernel_error OR rule.groups: system_error

Part 9: Automation and Response#

Active Response Configuration#

1
<ossec_config>
2
  <active-response>
3
    <command>disable-usb</command>
4
    <location>local</location>
5
    <rules_id>110006</rules_id>
6
    <timeout>0</timeout>
7
  </active-response>
8

9
  <active-response>
10
    <command>snapshot-memory</command>
11
    <location>local</location>
12
    <rules_id>110003</rules_id>
13
    <timeout>0</timeout>
14
  </active-response>
15
</ossec_config>

Custom Response Scripts#

Create /var/ossec/active-response/bin/disable-usb.sh:

1
#!/bin/bash
2
# Disable USB storage when suspicious device detected
3

4
ACTION=$1
5
USER=$2
6
IP=$3
7
ALERTID=$4
8
RULEID=$5
9

10
if [ "$ACTION" = "add" ]; then
11
    # Disable USB storage
12
    echo 'install usb-storage /bin/true' > /etc/modprobe.d/disable-usb-storage.conf
13

14
    # Log action
15
    logger -t wazuh-usb "USB storage disabled due to alert $ALERTID"
16

17
    # Remove existing USB storage module
18
    rmmod usb_storage 2>/dev/null
19
fi
20

21
exit 0

Part 10: Best Practices#

1. Storage Management#

1
# Implement log rotation
2
cat > /etc/logrotate.d/wazuh-kernel << EOF
3
/var/ossec/logs/archives/kernel-*.log {
4
    daily
5
    rotate 30
6
    compress
7
    delaycompress
8
    missingok
9
    notifempty
10
    create 0640 wazuh wazuh
11
}
12
EOF

2. Regular Maintenance#

1
#!/bin/bash
2
# Weekly kernel log analysis
3

4
# Check for unusual module activity
5
echo "=== Kernel Module Analysis ==="
6
lsmod | awk '{print $1}' | sort | comm -13 /etc/wazuh/baseline/modules.txt -
7

8
# Review high-severity kernel events
9
echo "=== High Severity Events (Last 7 days) ==="
10
grep -E "level\":(10|11|12|13|14|15)" /var/ossec/logs/alerts/alerts.json | \
11
    jq -r 'select(.rule.groups[]? | contains("kernel"))'
12

13
# Performance metrics
14
echo "=== Kernel Log Volume ==="
15
journalctl --since "7 days ago" _TRANSPORT=kernel | wc -l

3. Documentation#

Maintain a runbook for kernel security events:

1
## Kernel Security Event Response Playbook
2

3
### Event: Tainted Module Detected (Rule 110002)
4
1. Isolate affected system
5
2. Capture module information: `modinfo <module_name>`
6
3. Check module signature: `modprobe --dump-modversions <module>`
7
4. Review system logs for related activity
8
5. If confirmed malicious, remove and block
9

10
### Event: Memory Corruption (Rule 110003)
11
1. Identify affected process
12
2. Collect memory dump if possible
13
3. Review application logs
14
4. Check for known exploits
15
5. Apply patches or mitigations

Conclusion#

Kernel log monitoring with Wazuh provides deep visibility into system security and stability. By implementing the configurations in this guide, you’ll be able to:

🔍 Detect kernel-level attacks and rootkits
🛡️ Identify memory corruption and exploitation attempts
📊 Monitor system resource issues
🚨 Track hardware changes and USB devices
🔐 Enforce security policies at the kernel level

Remember: The kernel is the last line of defense. Monitor it well, and you’ll catch attacks that bypass user-space security.

Additional Resources#

Building robust security monitoring, one kernel message at a time. Stay vigilant! 🛡️

Newsletter