The eBPF Ecosystem: Tools, Frameworks, and Production Solutions
The eBPF ecosystem has evolved from a kernel feature to a rich collection of tools, frameworks, and production-ready solutions. This guide explores the entire ecosystem, helping you choose the right tools for your use case.
Tool Categories Overview
graph TD A[eBPF Tools] --> B[Development Tools] A --> C[Tracing & Profiling] A --> D[Networking] A --> E[Security] A --> F[Observability]
B --> G[bcc] B --> H[libbpf] B --> I[bpftrace]
C --> J[perf] C --> K[flamegraph]
D --> L[Cilium] D --> M[Calico eBPF] D --> N[Katran]
E --> O[Falco] E --> P[Tetragon]
F --> Q[Pixie] F --> R[Parca]
Core Development Tools
1. BCC (BPF Compiler Collection)
BCC makes eBPF programming accessible with Python and C++ frontends.
Installation
# Ubuntu/Debiansudo apt-get install bpfcc-tools linux-headers-$(uname -r)
# Fedorasudo dnf install bcc bcc-tools kernel-devel
# From sourcegit clone https://github.com/iovisor/bcc.git# Follow build instructions
Example: Process Execution Tracker
#!/usr/bin/pythonfrom bcc import BPF
# BPF programprog = """#include <uapi/linux/ptrace.h>#include <linux/sched.h>
struct data_t { u32 pid; u32 ppid; char comm[TASK_COMM_LEN]; char filename[256];};
BPF_PERF_OUTPUT(events);
int trace_exec(struct pt_regs *ctx, const char __user *filename, const char __user *const __user *__argv, const char __user *const __user *__envp) {
struct data_t data = {}; struct task_struct *task;
data.pid = bpf_get_current_pid_tgid() >> 32;
task = (struct task_struct *)bpf_get_current_task(); data.ppid = task->real_parent->tgid;
bpf_get_current_comm(&data.comm, sizeof(data.comm)); bpf_probe_read_user(&data.filename, sizeof(data.filename), filename);
events.perf_submit(ctx, &data, sizeof(data)); return 0;}"""
# Initialize BPFb = BPF(text=prog)b.attach_kprobe(event="__x64_sys_execve", fn_name="trace_exec")
# Process eventsdef print_event(cpu, data, size): event = b["events"].event(data) print(f"PID: {event.pid} PPID: {event.ppid} " f"COMM: {event.comm.decode()} " f"EXEC: {event.filename.decode()}")
# Loop with callbackb["events"].open_perf_buffer(print_event)while True: b.perf_buffer_poll()
BCC Tools Collection
# List available toolsls /usr/share/bcc/tools/
# Popular toolssudo biolatency # Block I/O latency histogramsudo biosnoop # Block I/O operationssudo cachestat # Page cache hit/miss statssudo execsnoop # Trace process executionsudo opensnoop # Trace open() syscallssudo tcpconnect # Trace TCP connectionssudo tcpretrans # TCP retransmissions
2. bpftrace - High-Level Tracing Language
bpftrace provides a domain-specific language for eBPF, inspired by DTrace and awk.
Installation
# Ubuntu/Debiansudo apt-get install bpftrace
# Build from source for latest featuresgit clone https://github.com/iovisor/bpftracecd bpftrace && mkdir build && cd buildcmake -DCMAKE_BUILD_TYPE=Release ..make -j$(nproc)sudo make install
One-Liner Examples
# Count syscalls by programsudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
# Show files opened by processsudo bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm, str(args->filename)); }'
# TCP accept() backlogsudo bpftrace -e 'kretprobe:inet_csk_accept { @[comm] = count(); }'
# Block I/O latencysudo bpftrace -e 'kprobe:blk_account_io_start { @start[arg0] = nsecs; } kprobe:blk_account_io_done /@start[arg0]/ { @usecs = hist((nsecs - @start[arg0]) / 1000); delete(@start[arg0]); }'
Complex Script Example
#!/usr/bin/env bpftrace
// trace_slowqueries.bt - MySQL slow query tracer
BEGIN { printf("Tracing MySQL queries slower than 100ms... Hit Ctrl-C to end.\n");}
uprobe:/usr/sbin/mysqld:*dispatch_command* { @query_start[tid] = nsecs; @query[tid] = str(arg2);}
uretprobe:/usr/sbin/mysqld:*dispatch_command* /@query_start[tid]/ { $duration_ms = (nsecs - @query_start[tid]) / 1000000;
if ($duration_ms > 100) { printf("%s: slow query (%d ms): %s\n", strftime("%H:%M:%S", nsecs), $duration_ms, @query[tid]); }
delete(@query_start[tid]); delete(@query[tid]);}
END { clear(@query_start); clear(@query);}
3. libbpf - Modern eBPF Development
libbpf is the official library for eBPF program development with CO-RE support.
Project Structure
my_project/├── src/│ ├── my_prog.bpf.c # eBPF program│ ├── my_prog.c # User-space code│ └── my_prog.h # Shared headers├── vmlinux.h # Generated kernel headers└── Makefile
Generating vmlinux.h
# Generate vmlinux.h for CO-REbpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
Modern eBPF Program with CO-RE
#include "vmlinux.h"#include <bpf/bpf_helpers.h>#include <bpf/bpf_tracing.h>#include <bpf/bpf_core_read.h>
char LICENSE[] SEC("license") = "Dual BSD/GPL";
struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 256 * 1024);} rb SEC(".maps");
struct event { u32 pid; u8 comm[16]; u64 mntns_id; u8 filename[256];};
SEC("tracepoint/syscalls/sys_enter_openat")int trace_openat_enter(struct trace_event_raw_sys_enter* ctx) { struct event *e; struct task_struct *task;
e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0); if (!e) return 0;
// Get PID and comm e->pid = bpf_get_current_pid_tgid() >> 32; bpf_get_current_comm(&e->comm, sizeof(e->comm));
// Get mount namespace ID using CO-RE task = (struct task_struct *)bpf_get_current_task(); BPF_CORE_READ_INTO(&e->mntns_id, task, nsproxy, mnt_ns, ns.inum);
// Read filename bpf_probe_read_user_str(&e->filename, sizeof(e->filename), (void *)ctx->args[1]);
bpf_ringbuf_submit(e, 0); return 0;}
Production Networking Solutions
1. Cilium - eBPF-based Networking and Security
Cilium provides networking, observability, and security for Kubernetes.
Installation
# Install Cilium CLIcurl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gzsudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
# Install on Kubernetescilium installcilium status --wait
Network Policy Example
apiVersion: cilium.io/v2kind: CiliumNetworkPolicymetadata: name: api-protectionspec: endpointSelector: matchLabels: app: api-server ingress: - fromEndpoints: - matchLabels: app: frontend toPorts: - ports: - port: "443" protocol: TCP rules: http: - method: "GET" path: "/api/v1/users" - method: "POST" path: "/api/v1/users" headers: - "Content-Type: application/json"
Hubble - Cilium’s Observability Layer
# Enable Hubblecilium hubble enable --ui
# Port forward to access UIcilium hubble ui
# CLI exampleshubble observe --namespace productionhubble observe --verdict DROPPEDhubble observe --http-status 5xx
2. Calico eBPF Mode
Calico offers eBPF dataplane for high-performance networking.
# Enable eBPF modekubectl patch felixconfiguration default --type='merge' \ -p '{"spec":{"bpfEnabled":true}}'
# Verify eBPF modekubectl get nodes -o yaml | grep 'projectcalico.org/bpfMode'
3. Katran - Facebook’s L4 Load Balancer
High-performance L4 load balancer using XDP.
// Simplified Katran logicSEC("xdp")int katran_lb(struct xdp_md *ctx) { void *data_end = (void *)(long)ctx->data_end; void *data = (void *)(long)ctx->data;
// Parse headers struct parsing_context pctx = {}; if (parse_packet(data, data_end, &pctx) < 0) return XDP_PASS;
// Lookup VIP struct vip_key vkey = { .address = pctx.dst, .port = pctx.port, .proto = pctx.proto };
struct vip_meta *vmeta = bpf_map_lookup_elem(&vip_map, &vkey); if (!vmeta) return XDP_PASS;
// Select real server __u32 hash = get_packet_hash(&pctx); __u32 real_idx = hash % vmeta->num_reals;
// Encapsulate and forward return encap_and_forward(ctx, real_idx);}
Security Tools
1. Falco - Runtime Security
Falco uses eBPF to detect anomalous behavior in applications.
Installation and Usage
# Install Falcocurl -s https://falco.org/repo/falcosecurity-3672BA8F.asc | apt-key add -echo "deb https://download.falco.org/packages/deb stable main" | \ tee -a /etc/apt/sources.list.d/falcosecurity.listapt-get update -yapt-get install -y falco
# Custom rulescat > /etc/falco/rules.d/custom.yaml <<EOF- rule: Unauthorized Process in Container desc: Detect unauthorized process execution in containers condition: > container and proc.name != "<authorized_processes>" and not proc.name in (known_system_procs) output: > Unauthorized process in container (user=%user.name command=%proc.cmdline container=%container.name) priority: WARNINGEOF
# Run Falco with eBPFfalco --modern-bpf
2. Tetragon - Runtime Security Enforcement
Tetragon provides eBPF-based security observability and enforcement.
# TracingPolicy exampleapiVersion: cilium.io/v1alpha1kind: TracingPolicymetadata: name: file-monitoringspec: kprobes: - call: "security_file_open" syscall: false args: - index: 0 type: "file" selectors: - matchArgs: - index: 0 operator: "Equal" values: - "/etc/passwd" - "/etc/shadow" matchActions: - action: Sigkill
Observability Platforms
1. Pixie - Instant Kubernetes Observability
# Install Pixiepx deploy
# Use Pixie CLIpx run px/cluster # Cluster overviewpx run px/http_data # HTTP traffic analysispx run px/pod # Pod resource usage
2. Parca - Continuous Profiling
# Deploy Parcakubectl apply -f https://github.com/parca-dev/parca/releases/latest/download/kubernetes-manifest.yaml
# Configure profilingapiVersion: parca.dev/v1alpha1kind: ParcaConfigmetadata: name: parca-configspec: mode: "pull" interval: "10s" profiles: - name: "cpu" path: "/debug/pprof/profile"
Specialized Tools
1. pwru - Packet Where Are You
Trace packet journey through the kernel:
# Install pwrugo install github.com/cilium/pwru@latest
# Trace packetssudo pwru --filter-dst-ip 10.0.0.1 --filter-dst-port 80
2. ecapture - SSL/TLS Text Capture
Capture SSL/TLS text without certificates:
# Capture OpenSSLsudo ecapture tls -i eth0 -w captured.pcap
# Capture specific processsudo ecapture tls -p 1234
3. kubectl-trace - eBPF Tracing for Kubernetes
# Installkubectl krew install trace
# Run bpftrace in podkubectl trace run pod/nginx -n default \ -e 'kprobe:do_sys_open { printf("%s opened %s\n", comm, str(arg1)); }'
Language Bindings and Frameworks
1. Go - ebpf-go
//go:generate go run github.com/cilium/ebpf/cmd/bpf2go counter counter.c
package main
import ( "log" "github.com/cilium/ebpf/link" "github.com/cilium/ebpf/rlimit")
func main() { // Remove memory limit if err := rlimit.RemoveMemlock(); err != nil { log.Fatal(err) }
// Load pre-compiled programs spec, err := loadCounter() if err != nil { log.Fatal(err) }
coll, err := ebpf.NewCollection(spec) if err != nil { log.Fatal(err) } defer coll.Close()
// Attach to tracepoint tp, err := link.Tracepoint("syscalls", "sys_enter_open", coll.Programs["count_open"]) if err != nil { log.Fatal(err) } defer tp.Close()}
2. Rust - aya
#![no_std]#![no_main]
use aya_bpf::{ macros::{kprobe, map}, maps::HashMap, programs::ProbeContext,};
#[map]static mut COUNTS: HashMap<u32, u64> = HashMap::with_max_entries(1024, 0);
#[kprobe(name="count_opens")]pub fn count_opens(ctx: ProbeContext) -> u32 { match try_count_opens(ctx) { Ok(ret) => ret, Err(ret) => ret, }}
fn try_count_opens(ctx: ProbeContext) -> Result<u32, u32> { let pid = ctx.pid();
unsafe { let count = COUNTS.get(&pid).unwrap_or(&0); COUNTS.insert(&pid, &(count + 1), 0)?; }
Ok(0)}
Tool Selection Guide
When to Use What?
Use Case | Tool | Why |
---|---|---|
Quick debugging | bpftrace | One-liners, rapid iteration |
Custom tools | BCC | Python convenience |
Production programs | libbpf | Performance, CO-RE |
Kubernetes networking | Cilium | Complete solution |
Security monitoring | Falco/Tetragon | Purpose-built |
Continuous profiling | Parca | Low-overhead profiling |
Load balancing | Katran/XDP | Line-rate performance |
Decision Tree
graph TD A[What do you need?] --> B{Type} B --> C[Debugging/Analysis] B --> D[Production System] B --> E[Security]
C --> F{Complexity} F --> G[Simple] --> H[bpftrace] F --> I[Complex] --> J[BCC]
D --> K{Domain} K --> L[Networking] --> M[Cilium/Calico] K --> N[Observability] --> O[Pixie/Parca] K --> P[Custom] --> Q[libbpf]
E --> R{Focus} R --> S[Detection] --> T[Falco] R --> U[Enforcement] --> V[Tetragon]
Best Practices
1. Development Workflow
# 1. Prototype with bpftracesudo bpftrace -e 'kprobe:vfs_read { @[comm] = count(); }'
# 2. Develop with BCCpython bcc_prototype.py
# 3. Production with libbpfmake production_ebpf
2. Testing Strategy
- Unit test eBPF programs with mock data
- Integration test with different kernel versions
- Load test for performance validation
- Security audit for privilege escalation
3. Monitoring eBPF Programs
# Monitor loaded programssudo bpftool prog list
# Check program statssudo bpftool prog profile id <id> duration 10
# Monitor map usagewatch -n 1 'sudo bpftool map dump id <id> | wc -l'
Future of eBPF Tools
Emerging Trends
- eBPF for Windows: Microsoft’s eBPF implementation
- Hardware Offload: SmartNIC eBPF support
- WASM Integration: Portable eBPF programs
- AI-Assisted Development: LLM-generated eBPF code
Upcoming Tools
- bpftime: User-space eBPF runtime
- eunomia-bpf: Simplified eBPF development
- Inspektor Gadget: Container-aware eBPF tools
Conclusion
The eBPF ecosystem offers tools for every use case, from quick debugging to production-grade infrastructure. Start with high-level tools like bpftrace, gradually move to frameworks like libbpf, and leverage production solutions like Cilium for specific domains.
The key is choosing the right tool for your needs and understanding the trade-offs between ease of use, performance, and flexibility.
Getting Started Resources
- Awesome eBPF - Curated list of eBPF resources
- eBPF.io - Official eBPF portal
- BCC Tutorial - BCC programming guide
- bpftrace Reference - Complete bpftrace guide
This completes our eBPF series! Start experimenting with these tools and join the growing eBPF community.