PCIe
Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti
Sissejuhatus
TODO
Tõõpõhimõte
- BAR - Base Address Register
Üle PCIe toimub nt cpu ja plokkseadme vahel andmevahetus selliselt
- tarkvara soovib lugeda faili sisu ja kasutab open/close/read/seek jms library call'isid
- operatsioonisüsteemi failisüsteemi osakond teisendab need eelmise punkti korraldused plokkseadme plokkide lugemisteks
- plokkseadme device driver moodustab eelmise punkti korraldustele vastavad nvme, scsi, sata protokolli korraldused
- device driver paigutab eelmises punktis moodustatud korraldused spetsiaalsesse mälu piirkonda (tõenäoliselt on see määratletud device driver kasutusse võtmisel)
- cpu tõlgendab teatud füüsilisi aadresse I/O aadressidena ja suunab tegevuse nt PCIe root-complex'ile
- pci riistvara saadab tegevused välja järjestikuliselt üle ühe või ename PCIe lane'i plokkseadme kontrollerile (nt NVMe kontroller)
- füüsilise plokkseadme kontroller tõlgendab ja edastab NVMe keelse käsu plokkseadmele endale (NVMe seadme puhul asub kontroller ja meedia jms koos ühes korpuses)
- füüsiline plokkseade täidab saabunud käsu lugedes andmed
- loetud andmed liiguvad vastupidist teeb pidi arvuti mällu kust neid kasutab protsessor
Väited
- device driver kasutusele võtmisel konfigureeritakse vastav plokkseadme kontroller ja seda kaudu tekib kontakt plokkseadme endaga
- device driver kasutusele võtmisel viiakse plokkseadme kontroller ja seda kaudu plokkseade nö olekusse, arvuti mälus registreeritakse nö struktuurid
- device driver poolt nö ettevalmistatud mälu aadressid on kernelile teada ja sinna kirjutamise ja lugemise abil saab edaspidi protsessor anda plokkseadmele korraldusi ning kirjutada ja lugeda andmeid
- PCIe protokolli peal kiht üleval pool sõidab NVMe protokoll
- device driver tarkuse abil hallatakse plokkseadme kontrollerit ja selle kaudu plokkseadet ennast
- cpu peal töötav tarkvara (tõenäoliselt on selleks device driver) räägib füüsilisele plokkseadmele vastavat protokolli (NVMe, SCSI, SATA)
Kasulikud lisamateralid
- https://www.quora.com/How-do-the-CPU-and-device-controller-communicate-for-example-an-HDD-controller
Proxmox keskkonnas QEMU kasutamine
Käsu põhiosa on vaadatud töötava vmid=104 virtuaalse arvuti jaoks 'ps aux | grep kvm' vms väljundist, sellele on juurde lisatud '-device' jms ridu
root@pm60-plms:~# cat start-104-orig.sh /usr/bin/kvm -id 104 -name tf-vm-1,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/104.qmp,server=on,wait=off -mon chardev=qmp,mode=control \ -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/104.pid -daemonize \ -smbios type=1,uuid=6a72218f-188e-494b-aeb4-7bcc71eca794 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults \ -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/104.vnc,password=on \ -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 4096 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \ -device vmgenid,guid=eb8ed186-bea3-4cce-8646-dbd472bc0b5e -device usb-tablet,id=tablet,bus=ehci.0,port=1 \ -chardev socket,id=serial0,path=/var/run/qemu-server/104.serial0,server=on,wait=off -device isa-serial,chardev=serial0 -device VGA,id=vga,bus=pcie.0,addr=0x1 \ -chardev socket,path=/var/run/qemu-server/104.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 \ -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on \ -iscsi initiator-name=iqn.1993-08.org.debian:01:dd493931668f \ -drive file=/dev/pve/vm-104-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 \ -netdev type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on \ -device virtio-net-pci,mac=FE:BC:F5:88:E0:79,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=1024 \ -netdev type=tap,id=net1,ifname=tap104i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on \ -device virtio-net-pci,mac=0A:A9:C9:F0:39:EF,netdev=net1,bus=pci.0,addr=0x13,id=net1,rx_queue_size=1024,tx_queue_size=1024 -machine type=q35+pve0 \ -device pcie-root-port,id=root_port1,chassis=1,slot=1,bus=pcie.0 \ -device x3130-upstream,id=upstream_port1,bus=root_port1 \ -device xio3130-downstream,id=downstream_port1,bus=upstream_port1,chassis=11,slot=11 \ -drive file=/dev/pve/vm-104-disk-1,if=none,id=drive-virtio1,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port1,drive=drive-virtio1,id=virtio1 \ -device xio3130-downstream,id=downstream_port2,bus=upstream_port1,chassis=12,slot=12 \ -drive file=/dev/pve/vm-104-disk-2,if=none,id=drive-virtio2,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port2,drive=drive-virtio2,id=virtio2 \ -device xio3130-downstream,id=downstream_port3,bus=upstream_port1,chassis=13,slot=13 \ -drive file=/dev/pve/vm-104-disk-3,if=none,id=drive-virtio3,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port3,drive=drive-virtio3,id=virtio3
kus
- TODO
Tulemusena moodustub selline virtuaalne nö riistvara
root@tf-vm-1:~# lspci -vt
-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
+-01.0 Device 1234:1111
+-02.0-[01-05]----00.0-[02-05]--+-00.0-[03]----00.0 Red Hat, Inc. Virtio block device
| +-01.0-[04]----00.0 Red Hat, Inc. Virtio block device
| \-02.0-[05]----00.0 Red Hat, Inc. Virtio block device
+-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4
+-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5
+-1a.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6
+-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2
+-1b.0 Intel Corporation 82801I (ICH9 Family) HD Audio Controller
+-1c.0-[06]--
+-1c.1-[07]--
+-1c.2-[08]--
+-1c.3-[09]--
+-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
+-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
+-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
+-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
+-1e.0-[0a-0e]--+-01.0-[0b]--+-03.0 Red Hat, Inc. Virtio memory balloon
| | +-08.0 Red Hat, Inc. Virtio console
| | +-0a.0 Red Hat, Inc. Virtio block device
| | +-12.0 Red Hat, Inc. Virtio network device
| | \-13.0 Red Hat, Inc. Virtio network device
| +-02.0-[0c]--
| +-03.0-[0d]--
| \-04.0-[0e]--
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
+-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
\-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller
kus
- TODO
ehk
root@tf-vm-1:~# lspci -nn 00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0] 00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) 00:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1a.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03) 00:1a.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03) 00:1a.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03) 00:1a.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03) 00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03) 00:1c.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.1 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.2 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.3 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1d.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03) 00:1d.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03) 00:1d.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03) 00:1d.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03) 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 92) 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02) 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02) 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02) 01:00.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Upstream) [104c:8232] (rev 02) 02:00.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 02:01.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 02:02.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 03:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 04:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 05:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 0a:01.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:03.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:04.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0b:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002] 0b:08.0 Communication controller [0780]: Red Hat, Inc. Virtio console [1af4:1003] 0b:0a.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1001] 0b:12.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000] 0b:13.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
kus
- TODO
ning lshw väljundis paistab muu hulgas
# lshw
..
*-pci
description: Host bridge
product: 82G33/G31/P35/P31 Express DRAM Controller
vendor: Intel Corporation
physical id: 100
bus info: pci@0000:00:00.0
version: 00
width: 32 bits
clock: 33MHz
*-display UNCLAIMED
description: VGA compatible controller
physical id: 1
bus info: pci@0000:00:01.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: vga_controller
configuration: latency=0
resources: memory:fa000000-faffffff memory:fea14000-fea14fff memory:c0000-dffff
*-pci:0
description: PCI bridge
product: QEMU PCIe Root port
vendor: Red Hat, Inc.
physical id: 2
bus info: pci@0000:00:02.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msix normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:22 memory:fea15000-fea15fff ioport:6000(size=16384) memory:fdc00000-fe1fffff ioport:fb800000(size=6291456)
*-pci
description: PCI bridge
product: XIO3130 PCI Express Switch (Upstream)
vendor: Texas Instruments
physical id: 0
bus info: pci@0000:01:00.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:29 ioport:6000(size=12288) memory:fdc00000-fe1fffff ioport:fb800000(size=6291456)
*-pci:0
description: PCI bridge
product: XIO3130 PCI Express Switch (Downstream)
vendor: Texas Instruments
physical id: 0
bus info: pci@0000:02:00.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:30 ioport:6000(size=4096) memory:fe000000-fe1fffff ioport:fbc00000(size=2097152)
*-scsi
description: SCSI storage controller
product: Virtio block device
vendor: Red Hat, Inc.
physical id: 0
bus info: pci@0000:03:00.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: scsi msix pm pciexpress bus_master cap_list
configuration: driver=virtio-pci latency=0
resources: irq:22 memory:fe000000-fe000fff memory:fbc00000-fbc03fff
*-virtio0
description: Virtual I/O device
physical id: 0
bus info: virtio@0
logical name: /dev/vda
size: 4GiB (4294MB)
configuration: driver=virtio_blk logicalsectorsize=512 sectorsize=512
...
kus
- scsi-storage-controlleriga tegeleb draiver virtio-pci (kui vaadata kogu lshw väljundit, siis on näha, et see draiver tegeleb lisaks veel muude nn virtio seadmetega, nt storage)
- pigem on see eripärane, et konkreetse plokkseadme juures on samuti märgitud draiver (virtio_blk)
PCIe seadme eemaldamine ja tagasi lisamine - variatsioon 1
Olgu eesmärgiks enp11s19 võrguliidese eemaldamine (mac 0a:a9:c9:f0:39:ef)
root@tf-vm-1:~# ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:c9:ef:5a:eb txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp11s19: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 0a:a9:c9:f0:39:ef txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.210.61 netmask 255.255.255.0 broadcast 192.168.210.255
inet6 fe80::fcbc:f5ff:fe88:e079 prefixlen 64 scopeid 0x20<link>
ether fe:bc:f5:88:e0:79 txqueuelen 1000 (Ethernet)
RX packets 3209 bytes 287410 (287.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2178 bytes 1225183 (1.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 98 bytes 8364 (8.3 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 98 bytes 8364 (8.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Selleks tuleb esmalt kindlaks teha pci seadme aadress, nt lshw väljundi vaatlemise teel
root@tf-vm-1:~# lshw | grep -A 2 -B 17 0a:a9:c9:f0:39:ef
*-network:1
description: Ethernet controller
product: Virtio network device
vendor: Red Hat, Inc.
physical id: 13
bus info: pci@0000:0b:13.0
version: 00
width: 64 bits
clock: 33MHz
capabilities: msix bus_master cap_list rom
configuration: driver=virtio-pci latency=0
resources: irq:20 ioport:4400(size=32) memory:fd883000-fd883fff memory:fb610000-fb613fff memory:fd840000-fd87ffff
*-virtio7 DISABLED
description: Ethernet interface
physical id: 0
bus info: virtio@7
logical name: enp11s19
serial: 0a:a9:c9:f0:39:ef
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=virtio_net driverversion=1.0.0 link=no multicast=yes
kus
- mac pci aadress on 0b:13.0
Seadme eemaldamiseks sobib öelda
root@tf-vm-1:~# echo 1 > /sys/bus/pci/devices/0000\:0b\:13.0/remove
Tulemusena lspci väljundist on seade kadunud. Seadme taas nähtavale toomiseks sobib temast pci-hierarhias kõrgema bridge-switchi juures öelda rescan, nt
root@tf-vm-1:~# echo 1 > /sys/bus/pci/devices/0000\:00\:1e.0/rescan
Tulemusena on seade tagasi lspci väljundis.
PCIe seadme eemaldamine ja tagasi lisamine - variatsioon 2
Eemaldamiseks, kuna virtio-pci driver tegeleb mitmete seadmetega, täpsustab echo sisu millise konkreetse seadme unbind toimub
root@tf-vm-1:~# echo -n "0000:0b:13.0" > /sys/bus/pci/drivers/virtio-pci/unbind
Tagasi toomiseks
root@tf-vm-1:~# echo -n "0000:0b:13.0" > /sys/bus/pci/drivers/virtio-pci/bind
