VPP - Vector Packet Processing

Sissejuhatus

TODO

Mõisted

• SPP - Scalar Packet Processing
• VPP - Vector Packet Processing
• FD.io - Fast Data input-output

Tööpõhimõte

Väited

• VPP lahendus on 'data-plane'
• VPP lahendus ei sisalda otseselt control-plain osakonda
• VPP lahendus sisaldab API liidest data plane juhtimiseks - vppctl on teatud mõttes control plane ja kohmakas
• VPP kasutuskohtadega tegelejad hoolitsevad control plane eest (nt netgate oma toodete puhul, linux-cp plugin abil on võimalik linux host pealt tegeleda)
• linux-cp - linux control plane - viitab vpp plugin abil saavutatud asjakorraldusele, et tavalisel viisil linux võrguseadistuste tegemine (net namespace 'dataplane' sees nö propageerub dataplane osakonda)

VPP lahenduse puhul toimuvad sündmused arvutis kolmes kohas

• nö vpp userspace osakonnas (kaasatud on ka dpdk, toimub võrgu mõttes kernel-bypass jms)
• tavaline kerneli osakond ja default võrgunduse namespace
• tavaline kerneli osakond ja non-default võrgunduse namespace nimega 'dataplane'

Linux CP

Nn linux-cp idee seisneb selles, et kerneli suhtes nö tavalises asukohas (aga eraldi võrgunduse namespace) toimuvale tehakse osa vpp dataplane peal olevat liiklust kättesaadavaks. Selle tulemusena

• 'tavalises asukohas' töötab nt ruuting daemon protsess bird, sinna ei toimi andmevahetus eriliselt kiiresti, aga seda pole ka vaja, sest ruutingu protokolle rääkivad asjaosalised teevad ruutingu muudatuste osas kokkuleppeid suhteliselt harva
• põhiline mass liiklust ei välju dataplane'ist (ja see on väga kiirest toimuv liiklus)
• vpp dataplane on täiesti eraldi nö linux kernelist
• vpp dataplane pealt nö välja toodud võrguliidesed asuvad eraldi linux kerneli namespace'is ('dataplane' nimeline)
• tavaline linux st mida bootloader boodib jms asub nö default namespace'is

Dünaamiline ruuting

Võrgujoonis

                               bird protess kinnitatud e1-0 nimelise võrguseadme külge - 10.6.13.131/24
  
                                    |            ___|___  10.8.13.0/24
                                ____|____       |___R3__|
                               |         |          |     10.7.13.5
                               |   R1    |----------|-------------------------------
                               |_________|  GigabitEthernet6/14/0 - 10.7.13.0/24
                                    |
                                    |  GigabitEthernet6/13/0 -> e1-0 - 10.6.13.131/24


                                 switch


                                    |  GigabitEthernet6/13/0 -> e1-0 - 10.6.13.132/24
                                ____|____
                               |         |  GigabitEthernet6/14/0 - 10.17.13.0/24
                               |   R2    |----------|--------------------------------
                               |_________|          |
                                    |            ___|___  10.17.13.5
                                    |           |___R4__|
                                    |               |     10.18.13.0/24

                               bird protess kinnitatud e1-0 nimelise võrguseadme külge - 10.6.13.131/24

kus

• bird on seadistatud kuulama vastavalt ühes ruuteris 10.6.13.131 ja teises ruuteris 10.6.13.132 aadressil (bird protsess töötab kerneli juures aga eraldi 'dataplane' nimelises namespace'is)
• enamus liiklust toimub ruuteri taga olevate võrkude vahel, nt paketid liiguvad 10.8.13.0/24 ja 10.18.13.0/24 võrkude vahel (pakettide endi src ega dst ip aadress ei ole 10.6.13.131/132); need paketid ei välju vpp dataplane osakonnast
• R1 ja R2 on suur-ruuterid, töötab bird ja kasutatakse dünaamilist ruutingut
• R3 ja R4 on väike-ruuterid, kasutatakse staatilist ruutingud

Staatiline ruuting

Staatilise ruutingu variatsioon lihtsalt illustreerib ühte võimalikku võrguliikluse katse korraldust

kus

• PVE virtuaalsete arvutite puhul tava-ruuting saavutab kiiruse 18 Gbit/s
• PVE virtuaalsete arvutite puhul VPP ruuting saavutab kiiruse 6 Gbit/s (natuke pettumus)

Paigaldamine - Proxmox virtuaalne arvuti

Väited

• VPP tarkvara paigaldatakse https://packagecloud.com/ repositooriumist
• 2024 aasta suvel on kõige värskem toetatud Ubuntu operatsioonisüsteemi versioon 22.04
• paigaldusjuhend asub aadressil https://s3-docs.fd.io/vpp/24.06/gettingstarted/installing/ubuntu.html
• secure boot parem välja lülitada, vastasel korral saab vpp paigaldamisel dmesg selliseid teateid

[  162.223607] Lockdown: vpp: raw io port access is restricted; see man kernel_lockdown.7

Virtuaalne arvuti peab toetama 2M suurust hugepagendust ja 1024 tükki, nt sobib sellise seadistusega virtuaalne arvuti

root@pve-moraal-x570:~# cat /etc/pve/qemu-server/9911.conf 
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: sn_srv_btrfs:9911/vm-9911-disk-0.raw,efitype=4m,pre-enrolled-keys=1,size=528K
ide2: none,media=cdrom
machine: q35
memory: 8192
meta: creation-qemu=7.1.0,ctime=1674411406
name: ubuntu-2204-vpp-01
net0: virtio=BC:24:11:9B:C3:E4,bridge=vmbr1,firewall=1,tag=111
numa: 1
ostype: l26
parent: enne-vpp-katset-puhas
scsihw: virtio-scsi-single
smbios1: uuid=174e8d69-5f6e-4296-8bd4-73e7247dae1e
sockets: 1
vga: virtio
virtio0: sn_srv_btrfs:9911/vm-9911-disk-1.raw,iothread=1,size=20G
virtio1: sn_srv_btrfs:9911/vm-9911-disk-2.raw,iothread=1,size=2G
vmgenid: 8de853d2-a9eb-4888-a3c6-b36d13896a96

ning kerneli rida

root@ubuntu-2204-vpp-01:~/20240813# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-5.15.0-118-generic root=/dev/mapper/system-root ro default_hugepagesz=2M hugepagesz=2M hugepages=2048

kus

• tundub, et numa: 1 ega kerneli hugepages osakond ei ole tegelikult vajalik
• host cpu tüüp on vajalik

apt repo seadistuse lisamiseks sobib avada https://packagecloud.io/fdio/release ning pressida nupul 'Debian', tulemusena pakutakse kopeerida selline käsk

# curl -s https://packagecloud.io/install/repositories/fdio/release/script.deb.sh | bash

käsu andmise tulemusena tekitatakse muu hulgas nt selline apt seadistus Ubuntu 22.04 puhul

# ls -ld /etc/apt/sources.list.d/fdio_release.list /etc/apt/keyrings/fdio_release-archive-keyring.gpg
-rw-r--r-- 1 root root 2823 Aug 12 22:26 /etc/apt/keyrings/fdio_release-archive-keyring.gpg
-rw-r--r-- 1 root root  357 Aug 12 22:26 /etc/apt/sources.list.d/fdio_release.list

# cat /etc/apt/sources.list.d/fdio_release.list                                                   

deb [signed-by=/etc/apt/keyrings/fdio_release-archive-keyring.gpg] https://packagecloud.io/fdio/release/ubuntu/ jammy main
deb-src [signed-by=/etc/apt/keyrings/fdio_release-archive-keyring.gpg] https://packagecloud.io/fdio/release/ubuntu/ jammy mai

vpp tarkvara põhiosa paigaldamiseks sobib öelda

# apt-get install vpp vpp-plugin-core vpp-plugin-dpdk

..
Setting up vpp (24.06-release) ...
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
sysctl: setting key "net.ipv4.conf.all.accept_source_route": Invalid argument
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/80-vpp.conf ...
vm.nr_hugepages = 1024
vm.max_map_count = 3096
vm.hugetlb_shm_group = 0
kernel.shmmax = 2147483648
* Applying /usr/lib/sysctl.d/99-protect-links.conf ...
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.conf ...
Created symlink /etc/systemd/system/multi-user.target.wants/vpp.service → /lib/systemd/system/vpp.service.
Setting up vpp-plugin-dpdk (24.06-release) ...
Setting up vpp-plugin-core (24.06-release) ...
...

Lisaks dpdk teemaga tegelemiseks driverctl utiliit

# apt-get install driverctl

Virtuaalse arvuti operatsioonisüsteemi ettevalmistamiseks, arvutis on kolm virtio tüüpi võrgukaart, kusjuures dpdk sisaldab virtio tuge

root@ubuntu-2204-vpp-01:~# driverctl list-devices network
0000:06:12.0 virtio-pci
0000:06:13.0 virtio-pci
0000:06:14.0 virtio-pci

root@ubuntu-2204-vpp-01:~# driverctl set-override 0000:06:13.0 vfio-pci
root@ubuntu-2204-vpp-01:~# driverctl set-override 0000:06:14.0 vfio-pci

Lisaks tuleks arvuti käivitada no-iommu režiimis

root@ubuntu-2204-vpp-01:~# grep iomm /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="vfio.enable_unsafe_noiommu_mode=1"

Tulemusena on arvutis sellised pci seadmed, võrguseadmed

root@ubuntu-2204-vpp-01:~# lspci -vvv | grep IOMM
	IOMMU group: 1
	IOMMU group: 0

Misc

/etc/vpp/startup.conf fail mõlemas arvutis

root@ubuntu-2204-vpp-01:~# less /etc/vpp/startup.conf
unix {
  nodaemon
  log /var/log/vpp/vpp.log
  full-coredump
  cli-listen /run/vpp/cli.sock
  gid vpp
  exec /etc/vpp/bootstrap.vpp
}

api-trace { on }
api-segment { gid vpp }
socksvr { default }

cpu {
        main-core 0
        corelist-workers 1-3
}

plugins {
        plugin default { disable }
        plugin dpdk_plugin.so { enable }
        plugin acl_plugin.so { enable }
        plugin linux_cp_plugin.so { enable }
        plugin linux_nl_plugin.so { enable }
}

logging {
   default-log-level debug
   default-syslog-log-level info
}

linux-cp { default netns dataplane }

bootstrap fail ühes ja teises arvutis

root@ubuntu-2204-vpp-01:~# cat /etc/vpp/bootstrap.vpp 
# set logging class linux-cp rate-limit 1000 level warn syslog-level notice

lcp default netns dataplane
lcp lcp-sync on
lcp lcp-auto-subint on

create loopback interface instance 0
lcp create loop0 host-if loop0
set interface state loop0 up
set interface ip address loop0 10.1.163.131/32
set interface ip address loop0 2999:678:d78::131/128

lcp create GigabitEthernet6/13/0 host-if e1-0
lcp create GigabitEthernet6/14/0 host-if e1-1

set interface mtu packet 1500 GigabitEthernet6/13/0
set interface ip address GigabitEthernet6/13/0 10.6.13.131/24
set interface state GigabitEthernet6/13/0 up

Teine arvuti

root@ubuntu-2204-vpp-02:~# less /etc/vpp/bootstrap.vpp 
# set logging class linux-cp rate-limit 1000 level warn syslog-level notice

lcp default netns dataplane
lcp lcp-sync on
lcp lcp-auto-subint on

create loopback interface instance 0
lcp create loop0 host-if loop0
set interface state loop0 up
set interface ip address loop0 10.1.163.132/32
set interface ip address loop0 2999:678:d78::132/128

lcp create GigabitEthernet6/13/0 host-if e1-0
lcp create GigabitEthernet6/14/0 host-if e1-1

set interface mtu packet 1500 GigabitEthernet6/13/0
set interface ip address GigabitEthernet6/13/0 10.6.13.132/24
set interface state GigabitEthernet6/13/0 up

Tulemuse esitamine

root@ubuntu-2204-vpp-01:~# ip netns exec dataplane ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
21: loop0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether de:ad:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: e1-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether bc:24:11:9b:c3:e5 brd ff:ff:ff:ff:ff:ff
23: e1-1: <BROADCAST,MULTICAST> mtu 9000 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether bc:24:11:9b:c3:e6 brd ff:ff:ff:ff:ff:ff

Operatsioonisüsteemi poolelt paistab koormus selline

kus

• kolm protsessorit millele seadistusfailis viidatakse on dpdk viisil koormatud ootuspäraselt

Käivitumisel logi

Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vpp[86714]: nl/nl: Added file 1
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vpp[86714]: nl/nl: Opened netlink socket 10
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: nl/nl: Added file 1
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: nl/nl: Opened netlink socket 10
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vpp[86714]: dpdk: EAL init args: --in-memory --no-telemetry --file-prefix vpp -a 0000:06:13.0 -a 0000:06:14.0
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL init args: --in-memory --no-telemetry --file-prefix vpp -a 0000:06:13.0 -a 0000:06:14.0 
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: vat_plugin_register: vmxnet3 plugin not loaded...
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: vat_plugin_register: avf plugin not loaded...
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: vat_plugin_register: dns plugin not loaded...
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: Loaded plugin: vpp_api_test_plugin.so
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: pot_vat_plugin_register: pot plugin not loaded...
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: Loaded plugin: vnet_ipsec_test_plugin.so
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: Loaded plugin: vnet_interface_test_plugin.so
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: Loaded plugin: vnet_arp_test_plugin.so
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: Loaded plugin: acl_test_plugin.so
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: vat_plugin_register: dhcp plugin not loaded...
...

Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: vat_plugin_register: geneve plugin not loaded...
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: vat-plug/load: vat_plugin_register: nsh plugin not loaded...
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Detected CPU lcores: 4
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Detected NUMA nodes: 1
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Detected static linkage of DPDK
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Selected IOVA mode 'PA'
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: No free 1048576 kB hugepages reported on node 0
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: VFIO support initialized
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Probe PCI driver: net_virtio (1af4:1000) device: 0000:06:13.0 (socket -1)
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Using IOMMU type 8 (No-IOMMU)
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Ignore mapping IO port bar(0)
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Probe PCI driver: net_virtio (1af4:1000) device: 0000:06:14.0 (socket -1)
Aug 17 17:57:10 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Ignore mapping IO port bar(0)
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: pair create: {loop0, tap3, loop0}
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: add: host:tap3 phy:loop0, host_if:loop0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: pair create: itf-pair: [0] loop0 tap3 loop0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: admin_state_change: itf-pair: [0] loop0 tap3 loop0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: sync_state: itf-pair: [0] loop0 tap3 loop0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: ip6/link: enable: loop0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: ip6/link: addr-add: loop0 -> 2001:678:d78::b
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: pair create: {GigabitEthernet6/13/0, tap1, e1-0}
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: add: host:tap1 phy:GigabitEthernet6/13/0, host_if:e1-0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: pair create: itf-pair: [1] GigabitEthernet6/13/0 tap1 e1-0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: pair create: {GigabitEthernet6/14/0, tap2, e1-1}
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: add: host:tap2 phy:GigabitEthernet6/14/0, host_if:e1-1
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: pair create: itf-pair: [2] GigabitEthernet6/14/0 tap2 e1-1
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: sync_state: itf-pair: [1] GigabitEthernet6/13/0 tap1 e1-0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: admin_state_change: itf-pair: [1] GigabitEthernet6/13/0 tap1 e1-0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: sync_state: itf-pair: [1] GigabitEthernet6/13/0 tap1 e1-0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/itf: set_interface_addr: itf-pair: [1] GigabitEthernet6/13/0 tap1 e1-0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: dpdk: Probe for interrupt mode for device GigabitEthernet6/13/0. Success.
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: dpdk: Interface GigabitEthernet6/13/0 started
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/router: Early message received for loop0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: message repeated 3 times: [ linux-cp/router: Early message received for loop0]
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: ip6/link: disable: loop0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: ip6/link: last-lock: loop0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/router: Early message received for GigabitEthernet6/13/0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: message repeated 3 times: [ linux-cp/router: Early message received for GigabitEthernet6/13/0]
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: dpdk: Interface GigabitEthernet6/13/0 stopped
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux-cp/router: Early message received for GigabitEthernet6/14/0
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: message repeated 3 times: [ linux-cp/router: Early message received for GigabitEthernet6/14/0]
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: dpdk: Probe for interrupt mode for device GigabitEthernet6/13/0. Success.
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: linux_epoll_file_update:120: epoll_ctl: File exists (errno 17)
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: dpdk: Interface GigabitEthernet6/13/0 started
Aug 17 17:57:12 ubuntu-2204-vpp-01 vpp[86714]: dpdk: EAL: Error disabling MSI-X interrupts for fd 21
Aug 17 17:57:14 ubuntu-2204-vpp-01 vpp[86714]: ip6/link: enable: loop0
Aug 17 17:57:14 ubuntu-2204-vpp-01 vpp[86714]: ip6/link: set-ll: loop0 -> fe80::dcad:ff:fe00:0
Aug 17 17:57:14 ubuntu-2204-vpp-01 vpp[86714]: ip6/link: enable: GigabitEthernet6/13/0
Aug 17 17:57:14 ubuntu-2204-vpp-01 vpp[86714]: ip6/link: set-ll: GigabitEthernet6/13/0 -> fe80::be24:11ff:fe9b:c3e5

kus

• TODO

acl

Tööpõhimõte

• esmalt tuleb kirjeldada ühe või teise index alla komplekt reegeleid
• reeglite komplekt seostatakse interface'i ja pakettide üle interface liikumise suunaga
• tuleb arvestada, et vaikimisi liiklust blokeeritakse

Nt reeglite komplekti moodustamine

vppctl set acl-plugin acl permit src 192.168.212.130/32 dst 192.168.213.132/32 proto 6 dport 22, \
permit src 192.168.212.130/32 dst 192.168.213.142/32 proto 6 dport 22, \
permit src 192.168.212.130/32 dst 192.168.213.152/32 proto 6 dport 22, \
permit src 192.168.212.130/32 dst 192.168.213.162/32 proto 6 dport 22, \
permit src 192.168.212.130/32 dst 192.168.213.172/32 proto 6 dport 22, \
permit src 192.168.212.130/32 dst 192.168.213.182/32 proto 0

Tulemusena

root@ubuntu-2204-vpp-01:~# vppctl show acl-plugin acl
acl-index 0 count 6 tag {cli}
          0: ipv4 permit src 192.168.212.130/32 dst 192.168.213.132/32 proto 6 sport 0-65535 dport 22
          1: ipv4 permit src 192.168.212.130/32 dst 192.168.213.142/32 proto 6 sport 0-65535 dport 22
          2: ipv4 permit src 192.168.212.130/32 dst 192.168.213.152/32 proto 6 sport 0-65535 dport 22
          3: ipv4 permit src 192.168.212.130/32 dst 192.168.213.162/32 proto 6 sport 0-65535 dport 22
          4: ipv4 permit src 192.168.212.130/32 dst 192.168.213.172/32 proto 6 sport 0-65535 dport 22
          5: ipv4 permit src 192.168.212.130/32 dst 192.168.213.182/32 proto 0 sport 0-65535 dport 0-65535

Komplekti seostamine liidese ja liikluse suunaga

root@ubuntu-2204-vpp-01:~# vppctl set acl-plugin interface GigabitEthernet6/13/0 input acl 0

root@ubuntu-2204-vpp-01:~# vppctl show acl-plugin acl
acl-index 0 count 6 tag {cli}
          0: ipv4 permit src 192.168.212.130/32 dst 192.168.213.132/32 proto 6 sport 0-65535 dport 22
          1: ipv4 permit src 192.168.212.130/32 dst 192.168.213.142/32 proto 6 sport 0-65535 dport 22
          2: ipv4 permit src 192.168.212.130/32 dst 192.168.213.152/32 proto 6 sport 0-65535 dport 22
          3: ipv4 permit src 192.168.212.130/32 dst 192.168.213.162/32 proto 6 sport 0-65535 dport 22
          4: ipv4 permit src 192.168.212.130/32 dst 192.168.213.172/32 proto 6 sport 0-65535 dport 22
          5: ipv4 permit src 192.168.212.130/32 dst 192.168.213.182/32 proto 0 sport 0-65535 dport 0-65535
  applied inbound on sw_if_index: 1
  used in lookup context index: 0

Olemasoleva komplekti sees muudatuste tegemiseks tuleb viidata komplektile index väärdusega, nt

vppctl set acl-plugin acl index 0 permit src 192.168.212.130/32 dst 192.168.213.132/32 proto 6 dport 22

Interface küljest reeglite lahti ühendamine

TODO

Komplekti kustutamine, ta ei tohi olla rakendatud inteface'ile

vpp# delete acl-plugin acl index 1
Deleted ACL index:1

Misc

Võrguliidese ip seadistus

vpp# show interface addr GigabitEthernet6/13/0
GigabitEthernet6/13/0 (up):
  L3 10.6.13.133/24

vpp# set interface ip address GigabitEthernet6/13/0 10.6.13.134/24
vpp# set interface ip address GigabitEthernet6/13/0 10.6.13.135/24

vpp# set interface ip address del GigabitEthernet6/13/0 10.6.13.135/24

Ruuting

vpp# ip route add 10.100.111.0/24 via 10.6.13.135
vpp# ip route del 10.100.111.0/24 via 10.6.13.135
vpp# show ip fib

Ruutingute automaatne jõudmine nö linuxist vpp osakonda

root@ubuntu-2204-vpp-01:~# for j in `seq 1 200`; do for i in `seq 1 200`; do ip netns exec dataplane ip route add 10.$j.$i.0/24 via 10.6.13.$i; done; done

root@ubuntu-2204-vpp-01:~# for j in `seq 1 200`; do for i in `seq 1 200`; do ip netns exec dataplane ip route del 10.$j.$i.0/24 via 10.6.13.$i; done; done

Tulemust on näha

root@ubuntu-2204-vpp-01:~# vppctl show ip fib | grep ^1 | nl
     1	10.6.13.0/32
     2	10.6.13.0/24
     3	10.6.13.131/32
     4	10.6.13.132/32
     5	10.6.13.134/32
     6	10.6.13.255/32
     7	194.1.163.35/32
     ...

Võrguliikluse kuulamisel namespace sees tuleb tcpdump utiliidi juures puhverdamisega tegelevat suvandit kasutada (-l)

root@ubuntu-2204-vpp-02:~# ip netns exec dataplane tcpdump -neli e1-0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on e1-0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:05:18.843096 bc:24:11:9b:c3:e5 > bc:24:11:ef:ef:67, ethertype IPv4 (0x0800), length 98: 10.6.13.131 > 10.6.13.132: ICMP echo request, id 41, seq 1, length 64
18:05:18.843117 bc:24:11:ef:ef:67 > bc:24:11:9b:c3:e5, ethertype IPv4 (0x0800), length 98: 10.6.13.132 > 10.6.13.131: ICMP echo reply, id 41, seq 1, length 64
18:05:19.861792 bc:24:11:9b:c3:e5 > bc:24:11:ef:ef:67, ethertype IPv4 (0x0800), length 98: 10.6.13.131 > 10.6.13.132: ICMP echo request, id 41, seq 2, length 64
18:05:19.861817 bc:24:11:ef:ef:67 > bc:24:11:9b:c3:e5, ethertype IPv4 (0x0800), length 98: 10.6.13.132 > 10.6.13.131: ICMP echo reply, id 41, seq 2, length 64

Probleemid

kerneli ja dataplane vahel seadistuste sync

• millegipärast ruutingud jõuavad automaatselt linux juurest vpp juurde, aga mitte vastupidi (praktiliselt ei ole see probleem)
• võrguliideste tekkimine-kadumine on süngis mõlemas suunas

vpp protsess ja net namespace

Mailing listis soovitatakse kasutada

root@ubuntu-2204-vpp-01:~# grep -B 2 NetworkNamespacePath /lib/systemd/system/vpp.service 
[Service]
Type=simple
NetworkNamespacePath=/var/run/netns/dataplane

root@ubuntu-2204-vpp-01:~# systemctl daemon-reload

ning samal ajal eemaldada vpp bootstrap ja seadistustest dataplane namespace viited

tulemusena

root@ubuntu-2204-vpp-01:~# lsns | grep vpp
4026532536 net         2 87839 root             /usr/bin/vpp -c /etc/vpp/startup.conf

root@ubuntu-2204-vpp-01:~# ifconfig -a | grep ^[a-z]
e1-0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
e1-1: flags=4098<BROADCAST,MULTICAST>  mtu 9000
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
loop0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000

root@ubuntu-2204-vpp-01:~# exit

Bird

TODO

Selleks, et bird protsess töötaks sobivas st dataplane nimelises namespace'is sobib tema service unit juures kasutada

..
NetworkNamespacePath=/var/run/netns/dataplane
...

Kasulikud lisamaterjalid

• 'how BIRD routes integrate into vpp' - https://lists.fd.io/g/vpp-dev/topic/how_bird_routes_integrate/90661567

Kasulikud lisamaterjalid

• 'lcp create: unknown input' - https://lists.fd.io/g/vpp-dev/topic/106289140#msg24512
• https://toonk.io/kernel-bypass-networking-with-fd-io-and-vpp/
• https://www.youtube.com/watch?v=KXM4waZ4HLI
• https://www.youtube.com/watch?v=hO2tlxURXJ0
• https://www.netgate.com/tnsr
• https://www.intel.com/content/www/us/en/developer/articles/technical/an-overview-of-advanced-server-based-networking-technologies.html
• https://www.youtube.com/watch?v=1Ta4aqVXxrA
• https://ipng.ch/s/articles/2021/08/12/vpp-linux-cp-part1/
• https://ipng.ch/s/articles/2024/02/10/vpp-on-freebsd-part-1/
• https://ersei.net/en/blog/vector-packet-processing
• https://en.wikipedia.org/wiki/Vector_Packet_Processing
• https://metebalci.com/blog/install-and-configure-dpdk-on-pcengines-apu4d4/
• https://medium.com/google-cloud/forwarding-over-100-mpps-with-fd-io-vpp-on-x86-62b9447da554
• https://github.com/ligato/vpp-base