Linux Kernel Performance Profiling, FlameGraphs, and eBPF

When systems experience unexplained latency spikes, CPU throttling, or lock contention, user-space APMs are often blind to kernel interactions. eBPF (Extended Berkeley Packet Filter) enables safe, sandboxed, high-performance tracing of kernel and application events in real time without kernel recompilation.

⚡ Quick Dive

Profiling Tools & SRE Use Cases

Tool / Technology Layer Primary Capability SRE Use Case
perf Linux Kernel CPU sampling, hardware counters, cache misses Profiling hot CPU functions & generating FlameGraphs
bpftrace Kernel / User High-level tracing language for eBPF Quick one-liner probes into syscalls and kernel latency
BCC Tools eBPF Collection Pre-built specialized CLI inspection tools biolatency (disk I/O), tcprtt (TCP latency), execsnoop
FlameGraphs Visualization Hierarchical stack-trace visualization Visualizing % of CPU time consumed per function

Essential bpftrace / BCC Commands

# 1. Trace block I/O latency histogram (detect slow disk writes)
sudo biolatency -m 10

# 2. Trace TCP connection latency (RTT) per remote host
sudo tcprtt -i 1

# 3. Trace new processes executed on the system in real time
sudo execsnoop

# 4. Measure time spent in read() syscall by process name
sudo bpftrace -e 'kprobe:sys_read { @start[tid] = nsecs; } kretprobe:sys_read /@start[tid]/ { @latency = hist(nsecs - @start[tid]); delete(@start[tid]); }'

📖 Extended Guide

1. Generating CPU FlameGraphs with perf

# 1. Capture stack traces at 99Hz on all CPUs for 30 seconds
sudo perf record -F 99 -a -g -- sleep 30

# 2. Collapse stack traces and render SVG FlameGraph
sudo perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > cpu-flamegraph.svg
  • On-CPU vs. Off-CPU Profiling:
    • On-CPU: Measures time spent executing instructions on CPU cores.
    • Off-CPU: Measures time spent blocked waiting for locks, disk I/O, database queries, or context switches.
⌂ All Guides