EBPF ja BCC utiliitide kasutamine

Sissejuhatus

TODO

Mõisted

• BCC - BPF Compiler Collection
• BPF - Berkeley Packet Filter
• BTF - BPF Type Format
• eBPF - extended Berkeley Packet Filter

Tööpõhimõte

TODO

Väited

• BCC kasutamiseks peavad olema süsteemis tuuma päised ja sellega kaasneb omajagu arendusvahendite paigaldust (gcc jms)

BCC skriptide komplektis on üldiselt kahte sorti utiliite

• esitatakse jooksvalt mingite tüüpi sündmuste toimumise kohta infot (nt execsnoop, opensnoop)
• esitatakse mingi aja jooksul toimunud mingit tüüpi sündmuste kohta statistikat (nt runqlen)

Paigaldamine

Ubuntu 16.04

Paigaldamist tutvustatakse aadressil https://github.com/iovisor/bcc/blob/master/INSTALL.md#ubuntu---binary

# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D4284CDD
# echo "deb https://repo.iovisor.org/apt/xenial xenial main" | sudo tee /etc/apt/sources.list.d/iovisor.list
# apt-get update
# apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

tulemusena

• paigaldatakse paketid

bcc-tools libbcc libbcc-examples python-bcc

• on /usr/share/bcc/tool kataloogitäis pyhton vms skripte

esimesel kasutamisel kirjutatakse dmesg'i

[N mai    3 07:27:05 2018] **********************************************************
[N mai    3 07:27:05 2018] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[N mai    3 07:27:05 2018] **                                                      **
[N mai    3 07:27:05 2018] ** trace_printk() being used. Allocating extra memory.  **
[N mai    3 07:27:05 2018] **                                                      **
[N mai    3 07:27:05 2018] ** This means that this is a DEBUG kernel and it is     **
[N mai    3 07:27:05 2018] ** unsafe for production use.                           **
[N mai    3 07:27:05 2018] **                                                      **
[N mai    3 07:27:05 2018] ** If you see this message and you are not debugging    **
[N mai    3 07:27:05 2018] ** the kernel, report this immediately to your vendor!  **
[N mai    3 07:27:05 2018] **                                                      **
[N mai    3 07:27:05 2018] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[N mai    3 07:27:05 2018] **********************************************************

Debian v. 9 Stretch

Paigaldada sõltuvused

# apt install linux-headers-amd64 sudo auditd build-essential cmake libllvm3.8 llvm-dev libclang-dev libelf-dev bison flex libedit-dev clang-format python python-netaddr python-pyroute2 luajit libluajit-5.1-dev arping iperf netperf ethtool devscripts zlib1g-dev libfl-dev

Kopeerida lähtetekst

# cd /var/tmp
# git clone https://github.com/iovisor/bcc.git

kus

• käesoleva teksti puhul tabati commiti

# git log --pretty=format:"%h - %an, %ar : %s" | head -n 4
dad0ad1 - yonghong-song, 2 days ago : Merge pull request #1729 from sandip4n/sandip4n_dev
2af81df - Sandipan Das, 2 days ago : Fix bpf_trace_printk() for big-endian targets
bd8f086 - yonghong-song, 2 days ago : Merge pull request #1727 from bobrik/runqlen-no-dup
0595f1b - Ivan Babrou, 3 days ago : Remove duplicate ebpf program load in tools/runqlen.py
...

• debian/control faili sõltuvuste osas muudatusi teha vaja pole.
• 2018 kevadel kohendada failis examples/networking/simulation.py, kommenteerides ns_ipdb.initdb() rea välja

         if out_ifc: out_ifc.up().commit()
        ns_ipdb.interfaces.lo.up().commit()
#        ns_ipdb.initdb()
        in_ifc = ns_ipdb.interfaces[in_ifname]

• kopeerida üle katkine test, et sellega tegelemine vahele jääks

# cp /var/tmp/bcc/tests/python/test_tools_memleak.py /var/tmp/bcc/tests/python/test_tools_smoke.py

Kompileerida

# debuild -b -uc -us

Tulemusena tekivad .deb paketid

# ls -ld *deb
-rw-r--r-- 1 root root   258592 May  5 10:38 bcc-lua_0.5.0-1_all.deb
-rw-r--r-- 1 root root   233928 May  5 10:38 bcc-tools_0.5.0-1_all.deb
-rw-r--r-- 1 root root 11878912 May  5 10:39 libbcc_0.5.0-1_amd64.deb
-rw-r--r-- 1 root root 10758006 May  5 10:38 libbcc-dbgsym_0.5.0-1_amd64.deb
-rw-r--r-- 1 root root   274252 May  5 10:38 libbcc-examples_0.5.0-1_amd64.deb
-rw-r--r-- 1 root root    22058 May  5 10:38 python3-bcc_0.5.0-1_all.deb
-rw-r--r-- 1 root root    21974 May  5 10:38 python-bcc_0.5.0-1_all.deb

Paigaldamiseks sobib öelda nt

# apt-get install ./bcc-tools_0.5.0-1_all.deb ./libbcc_0.5.0-1_amd64.deb ./python-bcc_0.5.0-1_all.deb

Paigaldamiseks arvutisse kus pole bcc tarkvara kompileeritud peab olema paigaldatud

# apt-get install linux-headers-amd64

Kasulikud lisamaterjalid

• https://github.com/iovisor/bcc/blob/master/INSTALL.md#debian---source

Debian v. 10 Buster

MÄRKUS 2018 kevad: ei õnnestunud kompileerida

Paigaldada sõltuvused

# apt install build-essential cmake libllvm4.0 llvm-dev libclang-dev libelf-dev bison flex libedit-dev clang-format python python-netaddr python-pyroute2 luajit libluajit-5.1-dev arping iperf netperf ethtool devscripts zlib1g-dev libfl-dev

muuta debian/control faili sõltuvuste osas

..
Build-Depends: debhelper (>= 9), cmake, libllvm4.0,
    llvm-dev, libclang-dev,
    libelf-dev, bison, flex, libfl-dev, libedit-dev, zlib1g-dev, git,
    clang-format, python (>= 2.7),
    python-netaddr, python-pyroute2, luajit, libluajit-5.1-dev, arping,
    inetutils-ping | iputils-ping, iperf, netperf, ethtool, devscripts,
    python3
Homepage: https://github.com/iovisor/bcc

...

Kompileerida

# debuild -b -uc -us

Paigaldada

# apt install ./bcc...

Kasulikud lisamaterjalid

• https://github.com/iovisor/bcc/blob/master/INSTALL.md#debian---source

Kasutamine

• filelife töötamise käigus moodustatud-kustutatud failide nimede jooksev esitamine

# /usr/share/bcc/tools/filelife

• mysql sql käskude esitamine

# /usr/share/bcc/tools/dbslower -x /usr/sbin/mysqld mysql -m 0
Tracing database queries for application /usr/sbin/mysqld slower than 0 ms...
TIME(s)        PID          MS QUERY
10.149637      643       1.203 select 1 + 1
20.429446      643       0.889 select version()
...

• postgresql sql käskude esitamine

TODO

• tcplife tcp ühenduste kestuse esitamiseks

# /usr/share/bcc/tools/tcplife -D 80
PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
1923  w3m        192.168.100.212 39362 217.146.71.187  80        0     0 16.80
1929  w3m        192.168.100.212 49178 85.222.234.14   80        0     2 107.51
...

Probleemid

• Ubuntu 16.04 tuum 4.4.0 juhtub vahel nii

# /usr/share/bcc/tools/opensnoop 
Killed
root@arvuti:/var/tmp# 
Broadcast message from systemd-journald@arvuti (Tue 2018-05-08 08:59:26 EEST):

systemd[1]: Caught <SEGV>, core dump failed (child 7960, code=killed, status=11/SEGV).

Broadcast message from systemd-journald@arvuti (Tue 2018-05-08 08:59:26 EEST):

systemd[1]: Freezing execution.

^C
root@arvuti:/var/tmp# dmesg 
-bash: ../sysdeps/nptl/fork.c:156: __libc_fork: Assertion `THREAD_GETMEM (self, tid) != ppid' failed.
Connection to 10.0.65.172 closed.
root@teine-arvuti# ssh root@10.0.65.172
ssh_exchange_identification: Connection closed by remote host

ning konsoolile kirjutatakse

Failed to send watchdog=1 notification: Connection refused

20260420 märkused

Väited

• bpf ja cgroups on tugevasti seotud tehnoloogiad
• cgroups ehk 'control groups' üks kontrolli realiseerimise vahend on bpf programmide kasutamine
• bpf programmid esinevad sõna otseses mõttes programmidena (nt device'ide puhul st kataloog /dev asjus), või programm ja andmed (nt võrgu puhul)

Tarkvara

# apt-get install libbpf-tools

cgroup ressursid

cgroup ressursside nimekirja puu kujuliseks esitamiseks

root@ph-minio-01:~# bpftool cgroup tree
CgroupPath
ID       AttachType      AttachFlags     Name
/sys/fs/cgroup/user.slice/user-0.slice/session-205.scope
    225      cgroup_inet_ingress multi           sd_fw_ingress
    224      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/user.slice/user-0.slice/session-275.scope
    265      cgroup_inet_ingress multi           sd_fw_ingress
    264      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/user.slice/user-0.slice/user@0.service
    223      cgroup_inet_ingress multi           sd_fw_ingress
    222      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/user.slice/user-0.slice/user@0.service/app.slice/nginx-06.service
    186      cgroup_inet_ingress multi           sd_fw_ingress
    185      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/user.slice/user-0.slice/user@0.service/app.slice/nginx-06.service/libpod-payload-8f94fa187611be402880d4988b287e246e88195b9f3ff3dd1dfebe277fd0b8f9
    187      cgroup_device   multi
/sys/fs/cgroup/user.slice/user-1000.slice/session-92.scope
    269      cgroup_inet_ingress multi           sd_fw_ingress
    268      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service
    271      cgroup_inet_ingress multi           sd_fw_ingress
    270      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/user.slice/user-1000.slice/session-278.scope
    273      cgroup_inet_ingress multi           sd_fw_ingress
    272      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/system.slice/systemd-udevd.service
    281      cgroup_inet_ingress multi           sd_fw_ingress
    280      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/system.slice/systemd-journald.service
    285      cgroup_inet_ingress multi           sd_fw_ingress
    284      cgroup_inet_egress multi           sd_fw_egress
    283      cgroup_device   multi           sd_devices
/sys/fs/cgroup/system.slice/systemd-machined.service
    279      cgroup_inet_ingress multi           sd_fw_ingress
    278      cgroup_inet_egress multi           sd_fw_egress
/sys/fs/cgroup/system.slice/rsyslog.service
    282      cgroup_device   multi           sd_devices
/sys/fs/cgroup/system.slice/systemd-timesyncd.service
    277      cgroup_device   multi           sd_devices
/sys/fs/cgroup/system.slice/systemd-logind.service
    276      cgroup_inet_ingress multi           sd_fw_ingress
    275      cgroup_inet_egress multi           sd_fw_egress
    274      cgroup_device   multi           sd_devices
root@ph-minio-01:~#

kus

• sd_devices
• sd_fw_ingress
• sd_fw_egress
• multi - ...

sisu küsimiseks

root@pm-varundus:~# bpftool prog show id 162
162: cgroup_device  name sd_devices  tag 654d7024997e7811  gpl run_time_ns 3666 run_cnt 12
	loaded_at 2026-04-20T18:23:31+0300  uid 0
	xlated 464B  jited 290B  memlock 4096B

root@pm-varundus:~# bpftool prog dump xlated id 162
   0: (61) r2 = *(u32 *)(r1 +0)
   1: (54) w2 &= 65535
   2: (61) r3 = *(u32 *)(r1 +0)
   3: (74) w3 >>= 16
   4: (61) r4 = *(u32 *)(r1 +4)
   5: (61) r5 = *(u32 *)(r1 +8)
   6: (55) if r2 != 0x2 goto pc+3
   7: (55) if r4 != 0x1 goto pc+2
   8: (55) if r5 != 0x3 goto pc+1
   9: (05) goto pc+46
...

Sündmused toimuvad sellises järjekorras

• namespace näitab protsessile erinevaid ressursside nimekirju
• control group takistab reaalselt tegevusi (nt ülemäära mälu kasutamine, lubamatusse kohta võrku pöördumine)

map sisu muutmine

abiks skript map sisu hex -> dec teisendamiseks

root@ph-minio-01:~# cat bpftool-map-dump.sh
#!/bin/bash

bpftool map dump id 138 | grep '^key' | while read -r line; do
    bytes=($(echo $line | cut -d: -f2))
    printf "Prefix: %d | IP: %d.%d.%d.%d\n" "$((16#${bytes[0]}))" "$((16#${bytes[4]}))" "$((16#${bytes[5]}))" "$((16#${bytes[6]}))" "$((16#${bytes[7]}))"
done

sd_fw_ingress sd_fw_egress juures map sisu muutmiseks

root@ph-minio-01:~# ./bpftool-map-dump.sh
Prefix: 32 | IP: 8.8.8.8
Prefix: 32 | IP: 127.0.0.1
Prefix: 24 | IP: 192.168.10.0

root@ph-minio-01:~# bpftool map delete id 51 key hex 20 00 00 00 08 08 08 08
root@ph-minio-01:~# bpftool map update id 51 key hex 20 00 00 00 09 09 09 09 value hex 01 00 00 00 00 00 00 00

root@ph-minio-01:~# ./bpftool-map-dump.sh
Prefix: 32 | IP: 9.9.9.9
Prefix: 32 | IP: 127.0.0.1
Prefix: 24 | IP: 192.168.10.0

Kasulikud lisamaterjalid

• https://jvns.ca/blog/2017/07/05/linux-tracing-systems/