PCIe
Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti
Sissejuhatus
TODO
Tõõpõhimõte
- BAR - Base Address Register
Üle PCIe toimub nt cpu ja plokkseadme vahel andmevahetus selliselt
- tarkvara soovib lugeda faili sisu ja kasutab open/close/read/seek jms library call'isid
- operatsioonisüsteemi failisüsteemi osakond teisendab need eelmise punkti korraldused plokkseadme plokkide lugemisteks
- plokkseadme device driver moodustab eelmise punkti korraldustele vastavad nvme, scsi, sata protokolli korraldused
- device driver paigutab eelmises punktis moodustatud korraldused spetsiaalsesse mälu piirkonda (tõenäoliselt on see määratletud device driver kasutusse võtmisel)
- cpu tõlgendab teatud füüsilisi aadresse I/O aadressidena ja suunab tegevuse nt PCIe root-complex'ile
- pci riistvara saadab tegevused välja järjestikuliselt üle ühe või ename PCIe lane'i plokkseadme kontrollerile (nt NVMe kontroller)
- füüsilise plokkseadme kontroller tõlgendab ja edastab NVMe keelse käsu plokkseadmele endale (NVMe seadme puhul asub kontroller ja meedia jms koos ühes korpuses)
- füüsiline plokkseade täidab saabunud käsu lugedes andmed
- loetud andmed liiguvad vastupidist teeb pidi arvuti mällu kust neid kasutab protsessor
Väited
- device driver kasutusele võtmisel konfigureeritakse vastav plokkseadme kontroller ja seda kaudu tekib kontakt plokkseadme endaga
- device driver kasutusele võtmisel viiakse plokkseadme kontroller ja seda kaudu plokkseade nö olekusse, arvuti mälus registreeritakse nö struktuurid
- device driver poolt nö ettevalmistatud mälu aadressid on kernelile teada ja sinna kirjutamise ja lugemise abil saab edaspidi protsessor anda plokkseadmele korraldusi ning kirjutada ja lugeda andmeid
- PCIe protokolli peal kiht üleval pool sõidab NVMe protokoll
- device driver tarkuse abil hallatakse plokkseadme kontrollerit ja selle kaudu plokkseadet ennast
- cpu peal töötav tarkvara (tõenäoliselt on selleks device driver) räägib füüsilisele plokkseadmele vastavat protokolli (NVMe, SCSI, SATA)
Kasulikud lisamateralid
- https://www.quora.com/How-do-the-CPU-and-device-controller-communicate-for-example-an-HDD-controller
Proxmox keskkonnas QEMU kasutamine
Käsu põhiosa on vaadatud töötava vmid=104 virtuaalse arvuti jaoks 'ps aux | grep kvm' vms väljundist, sellele on juurde lisatud '-device' jms ridu
root@pm60-plms:~# cat start-104-orig.sh /usr/bin/kvm -id 104 -name tf-vm-1,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/104.qmp,server=on,wait=off -mon chardev=qmp,mode=control \ -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/104.pid -daemonize \ -smbios type=1,uuid=6a72218f-188e-494b-aeb4-7bcc71eca794 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults \ -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/104.vnc,password=on \ -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 4096 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \ -device vmgenid,guid=eb8ed186-bea3-4cce-8646-dbd472bc0b5e -device usb-tablet,id=tablet,bus=ehci.0,port=1 \ -chardev socket,id=serial0,path=/var/run/qemu-server/104.serial0,server=on,wait=off -device isa-serial,chardev=serial0 -device VGA,id=vga,bus=pcie.0,addr=0x1 \ -chardev socket,path=/var/run/qemu-server/104.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 \ -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on \ -iscsi initiator-name=iqn.1993-08.org.debian:01:dd493931668f \ -drive file=/dev/pve/vm-104-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 \ -netdev type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on \ -device virtio-net-pci,mac=FE:BC:F5:88:E0:79,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=1024 \ -netdev type=tap,id=net1,ifname=tap104i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on \ -device virtio-net-pci,mac=0A:A9:C9:F0:39:EF,netdev=net1,bus=pci.0,addr=0x13,id=net1,rx_queue_size=1024,tx_queue_size=1024 -machine type=q35+pve0 \ -device pcie-root-port,id=root_port1,chassis=1,slot=1,bus=pcie.0 \ -device x3130-upstream,id=upstream_port1,bus=root_port1 \ -device xio3130-downstream,id=downstream_port1,bus=upstream_port1,chassis=11,slot=11 \ -drive file=/dev/pve/vm-104-disk-1,if=none,id=drive-virtio1,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port1,drive=drive-virtio1,id=virtio1 \ -device xio3130-downstream,id=downstream_port2,bus=upstream_port1,chassis=12,slot=12 \ -drive file=/dev/pve/vm-104-disk-2,if=none,id=drive-virtio2,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port2,drive=drive-virtio2,id=virtio2 \ -device xio3130-downstream,id=downstream_port3,bus=upstream_port1,chassis=13,slot=13 \ -drive file=/dev/pve/vm-104-disk-3,if=none,id=drive-virtio3,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port3,drive=drive-virtio3,id=virtio3
kus
- TODO
Tulemusena moodustub selline virtuaalne nö riistvara
root@tf-vm-1:~# lspci -vt
-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
+-01.0 Device 1234:1111
+-02.0-[01-05]----00.0-[02-05]--+-00.0-[03]----00.0 Red Hat, Inc. Virtio block device
| +-01.0-[04]----00.0 Red Hat, Inc. Virtio block device
| \-02.0-[05]----00.0 Red Hat, Inc. Virtio block device
+-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4
+-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5
+-1a.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6
+-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2
+-1b.0 Intel Corporation 82801I (ICH9 Family) HD Audio Controller
+-1c.0-[06]--
+-1c.1-[07]--
+-1c.2-[08]--
+-1c.3-[09]--
+-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
+-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
+-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
+-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
+-1e.0-[0a-0e]--+-01.0-[0b]--+-03.0 Red Hat, Inc. Virtio memory balloon
| | +-08.0 Red Hat, Inc. Virtio console
| | +-0a.0 Red Hat, Inc. Virtio block device
| | +-12.0 Red Hat, Inc. Virtio network device
| | \-13.0 Red Hat, Inc. Virtio network device
| +-02.0-[0c]--
| +-03.0-[0d]--
| \-04.0-[0e]--
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
+-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
\-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller
kus
- TODO
ehk
root@tf-vm-1:~# lspci -nn 00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0] 00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) 00:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1a.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03) 00:1a.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03) 00:1a.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03) 00:1a.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03) 00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03) 00:1c.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.1 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.2 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.3 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1d.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03) 00:1d.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03) 00:1d.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03) 00:1d.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03) 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 92) 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02) 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02) 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02) 01:00.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Upstream) [104c:8232] (rev 02) 02:00.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 02:01.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 02:02.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 03:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 04:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 05:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 0a:01.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:03.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:04.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0b:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002] 0b:08.0 Communication controller [0780]: Red Hat, Inc. Virtio console [1af4:1003] 0b:0a.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1001] 0b:12.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000] 0b:13.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
kus
- TODO
ning lshw väljundis paistab muu hulgas
# lshw
..
*-pci
description: Host bridge
product: 82G33/G31/P35/P31 Express DRAM Controller
vendor: Intel Corporation
physical id: 100
bus info: pci@0000:00:00.0
version: 00
width: 32 bits
clock: 33MHz
*-display UNCLAIMED
description: VGA compatible controller
physical id: 1
bus info: pci@0000:00:01.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: vga_controller
configuration: latency=0
resources: memory:fa000000-faffffff memory:fea14000-fea14fff memory:c0000-dffff
*-pci:0
description: PCI bridge
product: QEMU PCIe Root port
vendor: Red Hat, Inc.
physical id: 2
bus info: pci@0000:00:02.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msix normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:22 memory:fea15000-fea15fff ioport:6000(size=16384) memory:fdc00000-fe1fffff ioport:fb800000(size=6291456)
*-pci
description: PCI bridge
product: XIO3130 PCI Express Switch (Upstream)
vendor: Texas Instruments
physical id: 0
bus info: pci@0000:01:00.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:29 ioport:6000(size=12288) memory:fdc00000-fe1fffff ioport:fb800000(size=6291456)
*-pci:0
description: PCI bridge
product: XIO3130 PCI Express Switch (Downstream)
vendor: Texas Instruments
physical id: 0
bus info: pci@0000:02:00.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:30 ioport:6000(size=4096) memory:fe000000-fe1fffff ioport:fbc00000(size=2097152)
*-scsi
description: SCSI storage controller
product: Virtio block device
vendor: Red Hat, Inc.
physical id: 0
bus info: pci@0000:03:00.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: scsi msix pm pciexpress bus_master cap_list
configuration: driver=virtio-pci latency=0
resources: irq:22 memory:fe000000-fe000fff memory:fbc00000-fbc03fff
*-virtio0
description: Virtual I/O device
physical id: 0
bus info: virtio@0
logical name: /dev/vda
size: 4GiB (4294MB)
configuration: driver=virtio_blk logicalsectorsize=512 sectorsize=512
...
kus
- scsi-storage-controlleriga tegeleb draiver virtio-pci (kui vaadata kogu lshw väljundit, siis on näha, et see draiver tegeleb lisaks veel muude nn virtio seadmetega, nt storage)
- pigem on see eripärane, et konkreetse plokkseadme juures on samuti märgitud draiver (virtio_blk)
PCIe seadme eemaldamine ja tagasi lisamine - variatsioon 1
Olgu eesmärgiks enp11s19 võrguliidese eemaldamine (mac 0a:a9:c9:f0:39:ef)
root@tf-vm-1:~# ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:c9:ef:5a:eb txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp11s19: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 0a:a9:c9:f0:39:ef txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.210.61 netmask 255.255.255.0 broadcast 192.168.210.255
inet6 fe80::fcbc:f5ff:fe88:e079 prefixlen 64 scopeid 0x20<link>
ether fe:bc:f5:88:e0:79 txqueuelen 1000 (Ethernet)
RX packets 3209 bytes 287410 (287.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2178 bytes 1225183 (1.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 98 bytes 8364 (8.3 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 98 bytes 8364 (8.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Selleks tuleb esmalt kindlaks teha pci seadme aadress, nt lshw väljundi vaatlemise teel
root@tf-vm-1:~# lshw | grep -A 2 -B 17 0a:a9:c9:f0:39:ef
*-network:1
description: Ethernet controller
product: Virtio network device
vendor: Red Hat, Inc.
physical id: 13
bus info: pci@0000:0b:13.0
version: 00
width: 64 bits
clock: 33MHz
capabilities: msix bus_master cap_list rom
configuration: driver=virtio-pci latency=0
resources: irq:20 ioport:4400(size=32) memory:fd883000-fd883fff memory:fb610000-fb613fff memory:fd840000-fd87ffff
*-virtio7 DISABLED
description: Ethernet interface
physical id: 0
bus info: virtio@7
logical name: enp11s19
serial: 0a:a9:c9:f0:39:ef
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=virtio_net driverversion=1.0.0 link=no multicast=yes
kus
- mac pci aadress on 0b:13.0
Seadme eemaldamiseks sobib öelda
root@tf-vm-1:~# echo 1 > /sys/bus/pci/devices/0000\:0b\:13.0/remove
Tulemusena lspci väljundist on seade kadunud. Seadme taas nähtavale toomiseks sobib temast pci-hierarhias kõrgema bridge-switchi juures öelda rescan, nt
root@tf-vm-1:~# echo 1 > /sys/bus/pci/rescan
Tulemusena on seade tagasi lspci väljundis.
PCIe seadme eemaldamine ja tagasi lisamine - variatsioon 2
Eemaldamiseks, kuna virtio-pci driver tegeleb mitmete seadmetega, täpsustab echo sisu millise konkreetse seadme unbind toimub
root@tf-vm-1:~# echo -n "0000:0b:13.0" > /sys/bus/pci/drivers/virtio-pci/unbind
Tagasi toomiseks
root@tf-vm-1:~# echo -n "0000:0b:13.0" > /sys/bus/pci/drivers/virtio-pci/bind
bifurcation
Tegevuse eesmärk on leida võimalus hakata kasutama nö tavalises desktop moodi arvutis 2.5" formaadis nt U.3 liidesega nvme salvestusseadmeid, nt 4 tükki. Seejuures peaks arvestama, et naturaalselt kasutab üks nvme seade x4 st neli pcie lane'i. Ja nö tavalises desktop moodi arvutis on pce lane'isid nii 20 või 24 tükki. St peaks vaatama, et pcie lane'ide osas ei teki liigset konkurentsi nt videokaardi, võrgukaardi ning 'nvme ssd adapter' kaardi vahal. Tõenäoliselt konkurentsi olukorra jäävad seadmed siiski tööle, aga nad töötavad 'downgraded' režiimis, nt
root@pve-varundus-01:~# lspci -vvv | grep downgra LnkSta: Speed 8GT/s (downgraded), Width x4 (downgraded) LnkSta: Speed 8GT/s (downgraded), Width x4 (downgraded)
Tundub, et sellist nö laiendamist saab teha 'nvme ssd adapter' tüüpi kaardi abil, neid on kahesuguseid
- M.2 formaadis nö närimiskummi lehe kujuline seade - selle saab laienduskaardile otse külge kruvida - nt asus hyper m.2 seade
- kaabeldusega ühendatav tavaline server grade U.3 salvestusseade (nt Micron 7450 nvme seade) - tõenäoliselt nn Y kaabliga ühte kaardi porti saab kinnitada kaks NVME seadet
Väited
- Asjakorralduseks on vajalik, et emaplaat-chipset-bios toetab bifurcationit - see paistab välja bios PCI osakonnas x4/x4/x4/x4 jms valikute sättimisena.
- Lisakaardi valikul tuleb pöörata tähelepanu, kas asutakse kasutama 1. retimer, 2. redriver või 3. switch tüüpi lahendust
Kasulikud lisamaterjalid
- https://www.asus.com/support/faq/1037507/ - erinevate asus emaplaatide ja chipsettide bifurcation omadused
- asus pcie-to-nvme tooted - https://www.asus.com/bt/motherboards-components/motherboards/accessories/hyper-m-2-x16-card-v2/
- https://c-payne.com/
- https://www.supermicro.com/en/products/storage/cards
- https://www.supermicro.com/en/products/accessories/addon/AOC-SLG4-4E4T.php
- https://www.reddit.com/r/ASRock/comments/1caombg/can_anyone_confirm_x570_taichi_can_take_4_m2_pcie/ - millalgi siin sissekandes head inimesed arutlevad teema üle
- https://www.broadcom.com/products/storage/nvme-switch-adapter
