PCIe
Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti
Sissejuhatus
TODO
Kiirust tabel, GT/s on giga transactions, praktiliseelt Gbit/s moodi
- pcie v. 1.0 - 2.5 GT/s
- pcie v. 2.0 - 5 GT/s
- pcie v. 3.0 - 8 GT/s
- pcie v. 4.0 - 16 GT/s
- pcie v. 5.0 - 32 GT/s
Mõisted
- IDE (Integrity and Data Encryption) - uuemal ajal on initsiatiiv pcie kanalit krüptida
- BAR - Base Address Register
- AER - Advanced Error Reporting
Tõõpõhimõte
PCIe bridge vastutus
- oma portide vahel pcie pakette liigutada
- leppida kokku bridge ümber olevate seadmete adresseerimine
- leppida kokku portide küljes olevate seadmete kiirused
- korrigeerida vigu
- krüptida liiklust
- kontseptuaalselt on pcie bridge on sarnane ethernet network switch seadmele
Üle PCIe toimub nt cpu ja plokkseadme vahel andmevahetus selliselt
- tarkvara soovib lugeda faili sisu ja kasutab open/close/read/seek jms library call'isid
- operatsioonisüsteemi failisüsteemi osakond teisendab need eelmise punkti korraldused plokkseadme plokkide lugemisteks
- plokkseadme device driver moodustab eelmise punkti korraldustele vastavad nvme, scsi, sata protokolli korraldused
- device driver paigutab eelmises punktis moodustatud korraldused spetsiaalsesse mälu piirkonda (tõenäoliselt on see määratletud device driver kasutusse võtmisel)
- cpu tõlgendab teatud füüsilisi aadresse I/O aadressidena ja suunab tegevuse nt PCIe root-complex'ile
- pci riistvara saadab tegevused välja järjestikuliselt üle ühe või ename PCIe lane'i plokkseadme kontrollerile (nt NVMe kontroller)
- füüsilise plokkseadme kontroller tõlgendab ja edastab NVMe keelse käsu plokkseadmele endale (NVMe seadme puhul asub kontroller ja meedia jms koos ühes korpuses)
- füüsiline plokkseade täidab saabunud käsu lugedes andmed
- loetud andmed liiguvad vastupidist teeb pidi arvuti mällu kust neid kasutab protsessor
Väited
- device driver kasutusele võtmisel konfigureeritakse vastav plokkseadme kontroller ja seda kaudu tekib kontakt plokkseadme endaga
- device driver kasutusele võtmisel viiakse plokkseadme kontroller ja seda kaudu plokkseade nö olekusse, arvuti mälus registreeritakse nö struktuurid
- device driver poolt nö ettevalmistatud mälu aadressid on kernelile teada ja sinna kirjutamise ja lugemise abil saab edaspidi protsessor anda plokkseadmele korraldusi ning kirjutada ja lugeda andmeid
- PCIe protokolli peal kiht üleval pool sõidab NVMe protokoll
- device driver tarkuse abil hallatakse plokkseadme kontrollerit ja selle kaudu plokkseadet ennast
- cpu peal töötav tarkvara (tõenäoliselt on selleks device driver) räägib füüsilisele plokkseadmele vastavat protokolli (NVMe, SCSI, SATA)
Kasulikud lisamateralid
- https://www.quora.com/How-do-the-CPU-and-device-controller-communicate-for-example-an-HDD-controller
Proxmox keskkonnas QEMU kasutamine
Käsu põhiosa on vaadatud töötava vmid=104 virtuaalse arvuti jaoks 'ps aux | grep kvm' vms väljundist, sellele on juurde lisatud '-device' jms ridu
root@pm60-plms:~# cat start-104-orig.sh /usr/bin/kvm -id 104 -name tf-vm-1,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/104.qmp,server=on,wait=off -mon chardev=qmp,mode=control \ -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/104.pid -daemonize \ -smbios type=1,uuid=6a72218f-188e-494b-aeb4-7bcc71eca794 -smp 4,sockets=1,cores=4,maxcpus=4 -nodefaults \ -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/104.vnc,password=on \ -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 4096 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \ -device vmgenid,guid=eb8ed186-bea3-4cce-8646-dbd472bc0b5e -device usb-tablet,id=tablet,bus=ehci.0,port=1 \ -chardev socket,id=serial0,path=/var/run/qemu-server/104.serial0,server=on,wait=off -device isa-serial,chardev=serial0 -device VGA,id=vga,bus=pcie.0,addr=0x1 \ -chardev socket,path=/var/run/qemu-server/104.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 \ -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on \ -iscsi initiator-name=iqn.1993-08.org.debian:01:dd493931668f \ -drive file=/dev/pve/vm-104-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 \ -netdev type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on \ -device virtio-net-pci,mac=FE:BC:F5:88:E0:79,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=1024 \ -netdev type=tap,id=net1,ifname=tap104i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on \ -device virtio-net-pci,mac=0A:A9:C9:F0:39:EF,netdev=net1,bus=pci.0,addr=0x13,id=net1,rx_queue_size=1024,tx_queue_size=1024 -machine type=q35+pve0 \ -device pcie-root-port,id=root_port1,chassis=1,slot=1,bus=pcie.0 \ -device x3130-upstream,id=upstream_port1,bus=root_port1 \ -device xio3130-downstream,id=downstream_port1,bus=upstream_port1,chassis=11,slot=11 \ -drive file=/dev/pve/vm-104-disk-1,if=none,id=drive-virtio1,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port1,drive=drive-virtio1,id=virtio1 \ -device xio3130-downstream,id=downstream_port2,bus=upstream_port1,chassis=12,slot=12 \ -drive file=/dev/pve/vm-104-disk-2,if=none,id=drive-virtio2,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port2,drive=drive-virtio2,id=virtio2 \ -device xio3130-downstream,id=downstream_port3,bus=upstream_port1,chassis=13,slot=13 \ -drive file=/dev/pve/vm-104-disk-3,if=none,id=drive-virtio3,format=raw,cache=none,aio=native,detect-zeroes=on \ -device virtio-blk-pci,bus=downstream_port3,drive=drive-virtio3,id=virtio3
kus
- TODO
Tulemusena moodustub selline virtuaalne nö riistvara
root@tf-vm-1:~# lspci -vt
-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
+-01.0 Device 1234:1111
+-02.0-[01-05]----00.0-[02-05]--+-00.0-[03]----00.0 Red Hat, Inc. Virtio block device
| +-01.0-[04]----00.0 Red Hat, Inc. Virtio block device
| \-02.0-[05]----00.0 Red Hat, Inc. Virtio block device
+-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4
+-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5
+-1a.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6
+-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2
+-1b.0 Intel Corporation 82801I (ICH9 Family) HD Audio Controller
+-1c.0-[06]--
+-1c.1-[07]--
+-1c.2-[08]--
+-1c.3-[09]--
+-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
+-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
+-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
+-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
+-1e.0-[0a-0e]--+-01.0-[0b]--+-03.0 Red Hat, Inc. Virtio memory balloon
| | +-08.0 Red Hat, Inc. Virtio console
| | +-0a.0 Red Hat, Inc. Virtio block device
| | +-12.0 Red Hat, Inc. Virtio network device
| | \-13.0 Red Hat, Inc. Virtio network device
| +-02.0-[0c]--
| +-03.0-[0d]--
| \-04.0-[0e]--
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
+-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
\-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller
kus
- TODO
ehk
root@tf-vm-1:~# lspci -nn 00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0] 00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) 00:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1a.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03) 00:1a.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03) 00:1a.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03) 00:1a.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03) 00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03) 00:1c.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.1 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.2 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1c.3 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] 00:1d.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03) 00:1d.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03) 00:1d.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03) 00:1d.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03) 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 92) 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02) 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02) 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02) 01:00.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Upstream) [104c:8232] (rev 02) 02:00.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 02:01.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 02:02.0 PCI bridge [0604]: Texas Instruments XIO3130 PCI Express Switch (Downstream) [104c:8233] (rev 01) 03:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 04:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 05:00.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1042] (rev 01) 0a:01.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:03.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0a:04.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001] 0b:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002] 0b:08.0 Communication controller [0780]: Red Hat, Inc. Virtio console [1af4:1003] 0b:0a.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1001] 0b:12.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000] 0b:13.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
kus
- TODO
ning lshw väljundis paistab muu hulgas
# lshw
..
*-pci
description: Host bridge
product: 82G33/G31/P35/P31 Express DRAM Controller
vendor: Intel Corporation
physical id: 100
bus info: pci@0000:00:00.0
version: 00
width: 32 bits
clock: 33MHz
*-display UNCLAIMED
description: VGA compatible controller
physical id: 1
bus info: pci@0000:00:01.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: vga_controller
configuration: latency=0
resources: memory:fa000000-faffffff memory:fea14000-fea14fff memory:c0000-dffff
*-pci:0
description: PCI bridge
product: QEMU PCIe Root port
vendor: Red Hat, Inc.
physical id: 2
bus info: pci@0000:00:02.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msix normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:22 memory:fea15000-fea15fff ioport:6000(size=16384) memory:fdc00000-fe1fffff ioport:fb800000(size=6291456)
*-pci
description: PCI bridge
product: XIO3130 PCI Express Switch (Upstream)
vendor: Texas Instruments
physical id: 0
bus info: pci@0000:01:00.0
version: 02
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:29 ioport:6000(size=12288) memory:fdc00000-fe1fffff ioport:fb800000(size=6291456)
*-pci:0
description: PCI bridge
product: XIO3130 PCI Express Switch (Downstream)
vendor: Texas Instruments
physical id: 0
bus info: pci@0000:02:00.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:30 ioport:6000(size=4096) memory:fe000000-fe1fffff ioport:fbc00000(size=2097152)
*-scsi
description: SCSI storage controller
product: Virtio block device
vendor: Red Hat, Inc.
physical id: 0
bus info: pci@0000:03:00.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: scsi msix pm pciexpress bus_master cap_list
configuration: driver=virtio-pci latency=0
resources: irq:22 memory:fe000000-fe000fff memory:fbc00000-fbc03fff
*-virtio0
description: Virtual I/O device
physical id: 0
bus info: virtio@0
logical name: /dev/vda
size: 4GiB (4294MB)
configuration: driver=virtio_blk logicalsectorsize=512 sectorsize=512
...
kus
- scsi-storage-controlleriga tegeleb draiver virtio-pci (kui vaadata kogu lshw väljundit, siis on näha, et see draiver tegeleb lisaks veel muude nn virtio seadmetega, nt storage)
- pigem on see eripärane, et konkreetse plokkseadme juures on samuti märgitud draiver (virtio_blk)
PCIe seadme eemaldamine ja tagasi lisamine - variatsioon 1
Olgu eesmärgiks enp11s19 võrguliidese eemaldamine (mac 0a:a9:c9:f0:39:ef)
root@tf-vm-1:~# ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:c9:ef:5a:eb txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp11s19: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 0a:a9:c9:f0:39:ef txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.210.61 netmask 255.255.255.0 broadcast 192.168.210.255
inet6 fe80::fcbc:f5ff:fe88:e079 prefixlen 64 scopeid 0x20<link>
ether fe:bc:f5:88:e0:79 txqueuelen 1000 (Ethernet)
RX packets 3209 bytes 287410 (287.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2178 bytes 1225183 (1.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 98 bytes 8364 (8.3 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 98 bytes 8364 (8.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Selleks tuleb esmalt kindlaks teha pci seadme aadress, nt lshw väljundi vaatlemise teel
root@tf-vm-1:~# lshw | grep -A 2 -B 17 0a:a9:c9:f0:39:ef
*-network:1
description: Ethernet controller
product: Virtio network device
vendor: Red Hat, Inc.
physical id: 13
bus info: pci@0000:0b:13.0
version: 00
width: 64 bits
clock: 33MHz
capabilities: msix bus_master cap_list rom
configuration: driver=virtio-pci latency=0
resources: irq:20 ioport:4400(size=32) memory:fd883000-fd883fff memory:fb610000-fb613fff memory:fd840000-fd87ffff
*-virtio7 DISABLED
description: Ethernet interface
physical id: 0
bus info: virtio@7
logical name: enp11s19
serial: 0a:a9:c9:f0:39:ef
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=virtio_net driverversion=1.0.0 link=no multicast=yes
kus
- mac pci aadress on 0b:13.0
Seadme eemaldamiseks sobib öelda
root@tf-vm-1:~# echo 1 > /sys/bus/pci/devices/0000\:0b\:13.0/remove
Tulemusena lspci väljundist on seade kadunud. Seadme taas nähtavale toomiseks sobib temast pci-hierarhias kõrgema bridge-switchi juures öelda rescan, nt
root@tf-vm-1:~# echo 1 > /sys/bus/pci/rescan
Tulemusena on seade tagasi lspci väljundis.
PCIe seadme eemaldamine ja tagasi lisamine - variatsioon 2
Eemaldamiseks, kuna virtio-pci driver tegeleb mitmete seadmetega, täpsustab echo sisu millise konkreetse seadme unbind toimub
root@tf-vm-1:~# echo -n "0000:0b:13.0" > /sys/bus/pci/drivers/virtio-pci/unbind
Tagasi toomiseks
root@tf-vm-1:~# echo -n "0000:0b:13.0" > /sys/bus/pci/drivers/virtio-pci/bind
bifurcation
Tegevuse eesmärk on leida võimalus hakata kasutama nö tavalises desktop moodi arvutis 2.5" formaadis nt U.3 liidesega nvme salvestusseadmeid, nt 4 tükki. Seejuures peaks arvestama, et naturaalselt kasutab üks nvme seade x4 st neli pcie lane'i. Ja nö tavalises desktop moodi arvutis on pce lane'isid nii 20 või 24 tükki. St peaks vaatama, et pcie lane'ide osas ei teki liigset konkurentsi nt videokaardi, võrgukaardi ning 'nvme ssd adapter' kaardi vahal. Tõenäoliselt konkurentsi olukorra jäävad seadmed siiski tööle, aga nad töötavad 'downgraded' režiimis, nt
root@pve-varundus-01:~# lspci -vvv | grep downgra LnkSta: Speed 8GT/s (downgraded), Width x4 (downgraded) LnkSta: Speed 8GT/s (downgraded), Width x4 (downgraded)
Tundub, et sellist nö laiendamist saab teha 'nvme ssd adapter' tüüpi kaardi abil, neid on kahesuguseid
- M.2 formaadis nö närimiskummi lehe kujuline seade - selle saab laienduskaardile otse külge kruvida - nt asus hyper m.2 seade
- kaabeldusega ühendatav tavaline server grade U.3 salvestusseade (nt Micron 7450 nvme seade) - tõenäoliselt nn Y kaabliga ühte kaardi porti saab kinnitada kaks NVME seadet
Väited
- Asjakorralduseks on vajalik, et emaplaat-chipset-bios toetab bifurcationit - see paistab välja bios PCI osakonnas x4/x4/x4/x4 jms valikute sättimisena.
- Lisakaardi valikul tuleb pöörata tähelepanu, kas asutakse kasutama 1. retimer, 2. redriver või 3. switch tüüpi lahendust
Kasulikud lisamaterjalid
- https://www.asus.com/support/faq/1037507/ - erinevate asus emaplaatide ja chipsettide bifurcation omadused
- asus pcie-to-nvme tooted - https://www.asus.com/bt/motherboards-components/motherboards/accessories/hyper-m-2-x16-card-v2/
- https://c-payne.com/
- https://www.supermicro.com/en/products/storage/cards
- https://www.supermicro.com/en/products/accessories/addon/AOC-SLG4-4E4T.php
- https://www.reddit.com/r/ASRock/comments/1caombg/can_anyone_confirm_x570_taichi_can_take_4_m2_pcie/ - millalgi siin sissekandes head inimesed arutlevad teema üle
- https://www.broadcom.com/products/storage/nvme-switch-adapter
CXL
TODO
Kasulikud lisamaterjalid
- https://en.wikipedia.org/wiki/Compute_Express_Link
- https://www.gigabyte.com/PC-Accessory/AI-TOP-CXL-R5X4/
- https://www.youtube.com/watch?v=TGjrZdtgF04&t=339s
PCIe vigadega töötamine
PCIe Lane osakonnas esinevate vigade kohta info küsimine, nt
root@pve-svc-01:~# lspci -vvv | grep LaneErr LaneErrStat: 0 LaneErrStat: LaneErr at lane: 0 1 2 3 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: LaneErr at lane: 0 1 2 3 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: 0 LaneErrStat: LaneErr at lane: 0 1 2 3 4 5 6 7 LaneErrStat: 0 LaneErrStat: LaneErr at lane: 0 1 2 3 LaneErrStat: 0 LaneErrStat: 0
Muud
01:00.0 Non-Volatile memory controller: Micron Technology Inc 7450 PRO NVMe SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Micron Technology Inc 7450 PRO NVMe SSD
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 51
BIST result: 00
Region 0: Memory at fc140000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at fc100000 [disabled] [size=256K]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [b0] MSI-X: Enable+ Count=256 Masked-
Vector table: BAR=0 offset=00020000
PBA: BAR=0 offset=00023000
Capabilities: [c0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (downgraded), Width x4
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq- OBFF Via message, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [1b8 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: LaneErr at lane: 0 1 2 3
Capabilities: [920 v1] Lane Margining at the Receiver <?>
Capabilities: [9c0 v1] Physical Layer 16.0 GT/s <?>
Kernel driver in use: nvme
Kernel modules: nvme
kus
- UESta - uncorrectable errors
- CESta - correctable errors
Seadme poone pci kontakti võimekus
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <256ns, L1 unlimited
kus
- Speed 16 GT/s - ühe pcie lane kiirus - 16 giga transaction on iseloomulik pcie v. 4 standardile
- width x4 - kasutusel on 4 lane'i
- kokku on seadme ühenduse kiiruseks 4 x 16 = 64 GT/s ehk nii 6.4 GBait/s
Mis seadmel on süsteemiga kokku leppida õnnestunud
LnkSta: Speed 8GT/s (downgraded), Width x4
kus
- Speed 8 GT/s - ühe pci lane kiirus - 8 giga transaction on iseloomulik pcei v. 3 standardile
- with x4 - kasutusel on 4 lane'i
- kokku on seadme ühenduse kiiruseks 4 x 8 = 32 GT/s ehk nii 3.2 GBait/s
Probleemid pcie lane'idel
LaneErrStat: LaneErr at lane: 0 1 2 3
- probleemid esinevad kõigil seadme neljal lane'il
- selliste probleemide esitamine tekkis v. pcie 3.0 juures
PCIe nomenklatuur
pcie seadet identifitseeritakse kujuga 'xx:yy.z' ehk 'bus:device:function'.
Ühe reaalse ASRock Taichi x570 ja Ryzen 9 5900XT füüsilise arvuti koosseisu esitus
root@pve-svc-01:/zp_svc# lspci -vvvt
-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
+-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
+-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-01.2-[01-0d]----00.0-[02-0d]--+-01.0-[03]--+-00.0 NVIDIA Corporation GK208B [GeForce GT 710]
| | \-00.1 NVIDIA Corporation GK208 HDMI/DP Audio Controller
| +-02.0-[04]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
| +-03.0-[05-09]----00.0-[06-09]--+-03.0-[07]--
| | +-05.0-[08]----00.0 Intel Corporation I211 Gigabit Network Connection
| | \-07.0-[09]--
| +-04.0-[0a]----00.0 Intel Corporation 82574L Gigabit Network Connection
| +-08.0-[0b]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| | +-00.1 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| | \-00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| +-09.0-[0c]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
| \-0a.0-[0d]----00.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+-01.3-[0e]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
+-02.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-03.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-03.1-[0f]----00.0 Micron Technology Inc 7450 PRO NVMe SSD
+-03.2-[10]----00.0 Micron Technology Inc 7450 PRO NVMe SSD
+-03.3-[11]--+-00.0 Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
| \-00.1 Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
+-04.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-05.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-07.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-07.1-[12]----00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
+-08.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
+-08.1-[13]--+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| +-00.1 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
| +-00.3 Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| \-00.4 Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
+-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
+-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
+-18.0 Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0
+-18.1 Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1
+-18.2 Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2
+-18.3 Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3
+-18.4 Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4
+-18.5 Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5
+-18.6 Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6
\-18.7 Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7
kus
- 0000 - domeen - nn pc tüüpi arvutil on seda üks, serveritel võib olla mitu
- seadme nimi esineb pcie sõnavara kasutades kujul bus:device.function
- [ ] vahel esitatakse pordis oleva seadme abil tekitatud alam-bus'ide number (või numbrite vahemik)
Näite abil lugemine
-[0000:00]-+
..
+-01.2-[01-0d]----00.0-[02-0d]--+-01.0-[03]--+-00.0
...
- 01.2 tähistab asjaolu, et 'root pcie bridge' bus 00 küljes toimub midagi mida nimetakse seade '01' oma funktsiooniga '2' - xx.yy.z esitus on see '00:01.2' ja altpoolt saame teada et tegu on seadmega 'Starship/Matisse GPP Bridge'
- see 'midagi' sisaldab ise omakorda nelja bus'i - '01 - 0d'
- 'midagi' poolt moodustatud bus '01' peal on seade 00.0 - tema 'xx:yy.z' kujul olev nimi on '01.00.0' - altpoolt saame teada, et see on 'Matisse Switch'
- ta moodustab omakorda siinid '02 - 0d', millest esimese peal toimub midagi mida nimetatakse seade 01 ja funktsioon 0 - tema 'xx:yy.z' nimi on 02:01.0 - altpoolt saame teada, et see on 'Matisse PCIe GPP Bridge' (üks mitmetest)
- see seade moodustab omakorda ühe bus'i sisemiselt - '03' ja selle lüljes toimub midagi seadme 00 ja funktsioon 0 - '03:00.0' - altpoolt vaadates näeme, et 'GeForce GT 710'
Seoses CPU, mälu ja PCIe vahel (gemini google joonistus)
kus
- pcie root complex on nö otsekontaktis protsessori ja mäluga
Väited
- lspci -vvvt väljundist on selgesti näha, et pcie seadmed asuvad cpu suhtes kahes kohas: 1. otse protsessori külge kinnitatud, 2. nn chipset kaudu protsessori külge kinnitatud (sellisel juhul on komplekti peale kasutada 4 pcie lane'i)
pcie seadme absoluutse nime 'Fully Qualified PCI Address or the Segmented BDF Address' küsimiseks sobib öelda
root@pve-svc-01:~# lspci -PP -s 08:00.0 00:01.2/01:00.0/02:03.0/05:00.0/06:05.0/08:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
lspci ilma argumentideta esitab nimekirja
root@pve-svc-01:~# lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge 00:03.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 02:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 02:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 02:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 03:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1) 03:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1) 04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal] 05:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 06:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 06:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 06:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch 08:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 0a:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP 0b:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller 0b:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller 0c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) 0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) 0e:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal] 0f:00.0 Non-Volatile memory controller: Micron Technology Inc 7450 PRO NVMe SSD (rev 01) 10:00.0 Non-Volatile memory controller: Micron Technology Inc 7450 PRO NVMe SSD (rev 01) 11:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] 11:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function 13:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP 13:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP 13:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller 13:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
kus
- nö viimase astme bus väärtuse valib süsteem unikaalse - nii on võimalik unikaalselt väljendada seadmega seotud pcie aadressi kasutades viimast bus'i ning device ja function väärtust, nt '08:00.0'
- arvutis on palju pcie bridge'sid, mis moodustavad mitmekihilise hierarhia
- lscpi ilma argumentideta lühikese kujul väljundis näeb ka nende seadmete nimesid st peamiselt bus'id - bridge'd, mis 'lspci -vvvt' on ainult numbriga tähistatud
- nimekiri esitatakse numbrilises järjekorras, st mitte sisulise hierahria vms alusel
- huvitav oleks kasutada skripti, mis väljastab kaks arvu nö lõppkasutajat otseselt teenindavate seadmete kohta: 1. süsteemi pcie seadmete summaarse pcie lane count'i, 2. süsteemi pcie seadmete kasutuses oleva lane count'i - gemini google katsetas nii 6 korda, aga kasutatavat tulemust ei tekkinud
root@pve-svc-01:~# lspci | grep -vi Bridge | wc -l 22
