Wazuh Log Collection and Transmission - An In-Depth Technical Guide

Published: at 08:00 PM

Wazuh Log Collection and Transmission: An In-Depth Technical Guide

Overview of the Log Collection and Transmission Process

Wazuh is designed to ingest logs from multiple sources, including:

These various inputs are handled by the agent, which uses modular components to read, preprocess, and (optionally) compress the data before securely forwarding it to the manager. At the manager end, the logs are received, decompressed if needed, parsed, and analyzed.

How Logs Are Processed and Sent

1. Collection at the Agent

2. Optional Compression

3. Secure Transmission

4. Reception and Processing at the Manager

Architecture Diagrams

End-to-End Log Transmission Process

flowchart TD
    %% Log Sources
    subgraph Log_Sources["External Log Sources"]
        A["Log Files"]
        B["Syslog Streams"]
        C["Journald"]
    end

    %% Wazuh Agent Modules
    subgraph Agent["Wazuh Agent"]
        D["Log Collection Engine"]
        E["Preprocessing & Parsing"]
        F["Optional Compression"]
        G["Secure Transmission"]
    end

    %% Wazuh Manager Modules
    subgraph Manager["Wazuh Manager"]
        H["Network Listener"]
        I["Decompression & Parsing"]
        J["Analysis Engine"]
        K["Alerting & Reporting"]
    end

    %% Data Flow Connections
    A --> D
    B --> D
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I
    I --> J
    J --> K

Figure 1: End-to-end log collection and transmission process.

Runtime Interaction Sequence

sequenceDiagram
    participant L as Log Source
    participant A as Wazuh Agent
    participant M as Wazuh Manager
    participant R as Rule Engine

    L->>A: Generate Log Event
    A->>A: Collect & Preprocess Log
    A->>A: Compress Data (Optional)
    A->>M: Send Log over TLS
    M->>M: Receive & Decompress
    M->>R: Forward for Analysis
    R->>M: Return Analysis Results
    M->>M: Generate Alert (if needed)

Figure 2: Step-by-step sequence of log handling from agent to manager.

Code-Level Structure

classDiagram
    class LogCollector {
        +collectFromFile()
        +collectFromSyslog()
        +collectFromJournal()
    }

    class Preprocessor {
        +parseLogData()
        +structureData()
    }

    class CompressionModule {
        +compressData()
        +decompressData()
        -zlibRoutines
    }

    class SecureTransport {
        +establishTLSConnection()
        +sendData()
        +receiveData()
    }

    class AnalysisEngine {
        +applyRules()
        +generateAlerts()
    }

    LogCollector --> Preprocessor
    Preprocessor --> CompressionModule
    CompressionModule --> SecureTransport
    SecureTransport --> AnalysisEngine

Figure 3: Codebase modules involved in log processing and transmission in Wazuh.

Code-Level Insights and Libraries

Agent-Side Implementation

Manager-Side Implementation

For more detailed code-level insights, you can review the source files in the Wazuh GitHub repository v4.11.0. Searching for keywords like “compress”, “zlib”, or “socket” in the repository can reveal the exact implementation details.

Code Examples

Agent-Side Processing

Here’s a simplified example of how the agent might handle log collection and transmission:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zlib.h>
#include "tls_socket.h"  // Hypothetical TLS wrapper

// Compresses the log data using zlib
int compress_data(const char *input, size_t input_size, char *output, size_t *output_size) {
    return compress((Bytef *)output, output_size, (const Bytef *)input, input_size);
}

// Sends a log message from the agent to the manager
void send_log(const char* log_message) {
    char compressed_data[1024];
    size_t compressed_size = sizeof(compressed_data);

    // Compress the log message (if compression is enabled)
    if (compress_data(log_message, strlen(log_message), compressed_data, &compressed_size) != Z_OK) {
        fprintf(stderr, "Compression error\n");
        return;
    }

    // Connect to the manager via TLS
    TLS_Socket *socket = tls_connect("manager_address", 1514);
    if (socket == NULL) {
        fprintf(stderr, "Connection error\n");
        return;
    }

    // Send the compressed log data over the secure connection
    tls_send(socket, compressed_data, compressed_size);
    tls_close(socket);
}

Manager-Side Processing

And here’s how the manager might receive and process the logs:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zlib.h>
#include "tls_socket.h"  // Hypothetical TLS wrapper

// Decompresses the received data using zlib
int decompress_data(const char *input, size_t input_size, char *output, size_t *output_size) {
    return uncompress((Bytef *)output, output_size, (const Bytef *)input, input_size);
}

// Receives and processes a log message on the manager
void receive_log() {
    // Listen on a specific port for incoming TLS connections
    TLS_Socket *socket = tls_listen(1514);
    if (socket == NULL) {
        fprintf(stderr, "Listener error\n");
        return;
    }

    char buffer[1024];
    int bytes_received = tls_receive(socket, buffer, sizeof(buffer));
    tls_close(socket);

    // Decompress the log data if it was compressed
    char decompressed_data[2048];
    size_t decompressed_size = sizeof(decompressed_data);
    if (decompress_data(buffer, bytes_received, decompressed_data, &decompressed_size) != Z_OK) {
        fprintf(stderr, "Decompression error\n");
        return;
    }

    // Pass the decompressed data to the log parser/analysis module
    parse_log(decompressed_data, decompressed_size);
}

Conclusion

Wazuh’s log collection and transmission process is robust and optimized for both performance and security. By integrating optional compression (using libraries such as zlib) and employing secure transmission channels (via TLS), Wazuh ensures that even high volumes of log data are handled efficiently. The modular architecture and multiple-socket outputs further enhance its reliability, while the manager’s comprehensive analysis pipeline ensures that incoming logs are properly processed and actionable alerts are generated.

This technical deep-dive provides a detailed view of the internal mechanisms that power Wazuh’s log collection capabilities, from the initial data ingestion to the final analysis and alerting stages.

Performance Optimization Recommendations

  1. Enable Compression: For environments with high log volumes or limited bandwidth, ensure compression is enabled.
  2. Tune Buffer Sizes: Adjust buffer sizes in the agent configuration based on your log volume and network conditions.
  3. Multiple Managers: Consider setting up multiple managers for load distribution in large deployments.
  4. Optimize Log Collection: Configure appropriate intervals and file patterns to minimize unnecessary processing.
  5. Regular Maintenance: Keep both agents and manager components updated to benefit from performance improvements in newer versions.

By understanding and optimizing these mechanisms, Wazuh administrators can ensure efficient log collection and analysis across their infrastructure.