Visualizing Log Files with Netdata and Fluent Bit#

To visualize log files on multiple machines using Netdata and Fluent Bit, you’ll need to set up a data pipeline. Here’s a high-level overview of how you can achieve this:

Install Fluent Bit on each machine with log files
Configure Fluent Bit to collect and parse your log files
Set up Netdata on a central server or each machine
Configure Fluent Bit to send data to Netdata
Configure Netdata to receive and visualize the data from Fluent Bit

Let’s break this down into more detailed steps:

1. Install Fluent Bit#

Follow the official installation guide for your operating system: Fluent Bit Installation Guide

2. Configure Fluent Bit#

Create a configuration file (usually /etc/fluent-bit/fluent-bit.conf) to collect and parse your log files. Here’s a basic example:

1
[INPUT]
2
    Name tail
3
    Path /path/to/your/logfile.log
4
    Parser your_log_parser
5

6
[PARSER]
7
    Name your_log_parser
8
    Format regex
9
    Regex ^(?<time>[^ ]*) (?<message>.*)$
10
    Time_Key time
11
    Time_Format %Y-%m-%d %H:%M:%S
12

13
[OUTPUT]
14
    Name http
15
    Match *
16
    Host your_netdata_host
17
    Port 19999
18
    URI /api/v1/collector/charts
19
    Format json_stream

Adjust the Path, Parser, and Regex fields according to your log format.

3. Install Netdata#

Follow the official installation guide: Netdata Installation

4. Configure Netdata#

Enable the web_log plugin in Netdata by editing /etc/netdata/netdata.conf:

1
[web_log]
2
    enabled = yes

5. Start Both Services#

1
sudo systemctl start fluent-bit
2
sudo systemctl start netdata

Now, Fluent Bit should be collecting your log data and sending it to Netdata, which will visualize it in real-time.

Advanced Configuration#

This is a basic setup. Depending on your specific needs, you might want to add more advanced configurations, such as:

Filtering and Transforming Log Data#

Add filters to process your logs before sending them:

1
[FILTER]
2
    Name grep
3
    Match *
4
    Regex message error|warning|critical
5

6
[FILTER]
7
    Name record_modifier
8
    Match *
9
    Record hostname ${HOSTNAME}
10
    Record service_name my_application

Multiple Log Sources#

Monitor multiple log files by adding more INPUT sections:

1
[INPUT]
2
    Name tail
3
    Path /var/log/app1/*.log
4
    Tag app1
5
    Parser app1_parser
6

7
[INPUT]
8
    Name tail
9
    Path /var/log/app2/*.log
10
    Tag app2
11
    Parser app2_parser

Aggregation for Multiple Machines#

If you have multiple machines, you can set up a central Fluent Bit aggregator:

1
# On each machine
2
[OUTPUT]
3
    Name forward
4
    Match *
5
    Host central_fluent_bit_host
6
    Port 24224
7

8
# On central aggregator
9
[INPUT]
10
    Name forward
11
    Port 24224
12

13
[OUTPUT]
14
    Name http
15
    Match *
16
    Host netdata_host
17
    Port 19999
18
    URI /api/v1/collector/charts
19
    Format json_stream

Custom Parsers#

Create custom parsers for your specific log formats:

1
[PARSER]
2
    Name apache_access
3
    Format regex
4
    Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
5
    Time_Key time
6
    Time_Format %d/%b/%Y:%H:%M:%S %z
7

8
[PARSER]
9
    Name json_parser
10
    Format json
11
    Time_Key timestamp
12
    Time_Format %Y-%m-%dT%H:%M:%S.%L

Security Considerations#

When setting up this pipeline, consider:

TLS/SSL Encryption: Use HTTPS for the HTTP output
Authentication: Add authentication headers if required
Network Security: Ensure proper firewall rules between machines
Log Rotation: Configure log rotation to prevent disk space issues

Netdata Configuration for Better Visualization#

Configure Netdata to better visualize your log data:

1
# In /etc/netdata/go.d/web_log.conf
2
jobs:
3
  - name: custom_app_logs
4
    path: /var/log/custom_app/*.log
5
    custom_log_format:
6
      pattern: '(?P<address>[\da-f.:]+) - (?P<user>.*) \[(?P<time>.*)\] "(?P<method>[A-Z]+) (?P<url>.*) HTTP/[0-9.]+" (?P<code>[0-9]+) (?P<bytes_sent>[0-9]+) "(?P<referer>.*)" "(?P<user_agent>.*)"'
7
      time_format: "%d/%b/%Y:%H:%M:%S %z"

Alerting Configuration#

Set up alerts in Netdata based on log patterns:

1
# In /etc/netdata/health.d/logs.conf
2
alarm: high_error_rate
3
on: web_log.custom_app_logs
4
lookup: sum -5m unaligned of errors
5
units: errors
6
every: 1m
7
warn: $this > 100
8
crit: $this > 500
9
info: High error rate detected in application logs

Troubleshooting#

Common issues and solutions:

Fluent Bit not sending data: Check connectivity and firewall rules
Parser not matching: Test your regex patterns with sample log lines
High memory usage: Adjust buffer sizes and flush intervals
Missing data in Netdata: Verify the API endpoint and data format

This setup provides a robust solution for centralizing and visualizing logs from multiple machines using Netdata and Fluent Bit.