BPF Superpowers for Linux

  |   Source

A powerful set of tools is helpful not only for developers. On GNU/Linux strace is one such tool and helped me already with a wide variety of problems. However it is always confined to one process or a process tree and cannot help with system wide problems / questions. Although there have been many attempts in the past, the Linux developers seem to have found a common underlying infrastructure in the form of BPF for such tools, allowing flexible and high-performance probing.

Already back in 2016, the Netflix developer Brendan Gregg showed what is possible in his talk Linux BPF Superpowers.

Linux bcc/BPF Tracing Tools

If you want to get a quick idea of what is possible with this tool set, you can do this easily on a Debian Buster GNU/Linux system:

dzu@krikkit:~$ sudo apt install bpfcc-tools
[sudo] password for dzu:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ieee-data libbpfcc python-bpfcc python-netaddr
Suggested packages:
  ipython python-netaddr-docs
The following NEW packages will be installed:
  bpfcc-tools ieee-data libbpfcc python-bpfcc python-netaddr
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 15.9 MB of archives.
After this operation, 66.7 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian buster/main amd64 libbpfcc amd64 0.8.0-4 [13.4 MB]
Get:2 http://deb.debian.org/debian buster/main amd64 python-bpfcc all 0.8.0-4 [29.4 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 ieee-data all 20180805.1 [1590 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 python-netaddr all 0.7.19-1 [229 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 bpfcc-tools all 0.8.0-4 [654 kB]
Fetched 15.9 MB in 1s (11.4 MB/s)
Selecting previously unselected package libbpfcc.
(Reading database ... 307420 files and directories currently installed.)
Preparing to unpack .../libbpfcc_0.8.0-4_amd64.deb ...
Unpacking libbpfcc (0.8.0-4) ...
Selecting previously unselected package python-bpfcc.
Preparing to unpack .../python-bpfcc_0.8.0-4_all.deb ...
Unpacking python-bpfcc (0.8.0-4) ...
Selecting previously unselected package ieee-data.
Preparing to unpack .../ieee-data_20180805.1_all.deb ...
Unpacking ieee-data (20180805.1) ...
Selecting previously unselected package python-netaddr.
Preparing to unpack .../python-netaddr_0.7.19-1_all.deb ...
Unpacking python-netaddr (0.7.19-1) ...
Selecting previously unselected package bpfcc-tools.
Preparing to unpack .../bpfcc-tools_0.8.0-4_all.deb ...
Unpacking bpfcc-tools (0.8.0-4) ...
Setting up ieee-data (20180805.1) ...
Setting up python-netaddr (0.7.19-1) ...
Setting up libbpfcc (0.8.0-4) ...
Setting up python-bpfcc (0.8.0-4) ...
Setting up bpfcc-tools (0.8.0-4) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...
dzu@krikkit:~$

One simple but very powerful tool is execsnoop showing every exec call on the system. On Debian the package maintainer has decided to name all the tools with an -bpfcc suffix not to collide with tools from the perf suite, so we have to call it like this:

dzu@krikkit:~$ sudo execsnoop-bpfcc
PCOMM            PID    PPID   RET ARGS
nikola           11710  7479     0 /home/dzu/src/python/nikola/bin/nikola build
ldconfig         11711  11710    0 /sbin/ldconfig -p
sh               11715  11714    0 /bin/sh -c command -v debian-sa1 > /dev/null && debian-sa1 1 1
debian-sa1       11716  11715    0 /usr/lib/sysstat/debian-sa1 1 1
nikola           11717  7479     0 /home/dzu/src/python/nikola/bin/nikola build
ldconfig         11718  11717    0 /sbin/ldconfig -p
nikola           11724  7479     0 /home/dzu/src/python/nikola/bin/nikola build
ldconfig         11725  11724    0 /sbin/ldconfig -p
sshd             11741  797      0 /usr/sbin/sshd -D -R
sshd             11766  797      0 /usr/sbin/sshd -D -R
sh               11768  767      0 /bin/sh -c iptables -w -n -L INPUT | grep -q 'f2b-sshd[ \t]'
iptables         11769  11768    0 /usr/sbin/iptables -w -n -L INPUT
grep             11770  11768    0 /usr/bin/grep -q f2b-sshd[ \t]
sh               11771  767      0 /bin/sh -c iptables -w -I f2b-sshd 1 -s 61.177.172.158 -j REJECT --reject-with icmp-port-unreachable
iptables         11772  11771    0 /usr/sbin/iptables -w -I f2b-sshd 1 -s 61.177.172.158 -j REJECT --reject-with icmp-port-unreachable
nikola           11788  7479     0 /home/dzu/src/python/nikola/bin/nikola build
ldconfig         11789  11788    0 /sbin/ldconfig -p

This run shows the blog software Nikola building the page during my edits, some unsolicited attempts at logging in to my PC and fail2ban finally kicking in to block the IP address of the offending ssh attempt.

The nifty little tool is in fact a readable Python script dynamically loading C code to hook the syscall__execve and do_ret_sys_execve Linux kernel functions. This is indeed very elegant. It has to be kept in mind however that the C code will not get compiled into machine code but into byte code for the BPF virtual machine. This virtual machine is intentionally not a generic virtual machine to preserve safety and security guarantees.

Other interesting tools are biotop and cachetop. The first shows Block I/O operations whereas the latter shows Linux Page Cache hits / misses - both in a top like fashion.

If you are interested in BPF and you are a fan of printed books then you will be glad to hear that O'Reilly just published Linux Observability with BPF for your perusal.

All in all, BPF is an exciting new Linux kernel technology and it is interesting to reflect on how this subsystem originating as a "packet filter" has gained (and will gain) completely new applications. It certainly is a powerful tool for Linux developers.

Comments powered by Disqus