Mellanox ConnectX-6 Lx EN: erinevus redaktsioonide vahel
Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti
| 97. rida: | 97. rida: | ||
===MLNX EN tarkvara kasutamine=== |
===MLNX EN tarkvara kasutamine=== |
||
| + | |||
| + | ====Mõisted==== |
||
| + | |||
| + | * MFT - NVIDIA Firmware Tools, tõenäoliselt algupäraselt Mellanox Firmware Tools |
||
| + | |||
| + | ====Kasutamine füüsilise seadme tervikuna passthru režiimis==== |
||
| + | |||
| + | Väited |
||
| + | |||
| + | * üldiselt proxmox v. 8 keskkonnas saab mellanox seadme anda üle virtuaalsele arvuti tavalisel viisil (valides pve webgui liidses 'Add -> PCI device' ja näidates esimese MLNX seadme; teine lisatakse automaatselt |
||
| + | * tundub, et füüsilist mellanox seadet ei saa tervikuna nö täiuslikult virtuaalsele arvutile edasi anda, ühe asjana puudub jääb sr-iov võimekus |
||
| + | * virtuaalses arvutis saab võrguseadet kasutada tema PF osas, VF ei ole ligipääsetav |
||
| + | * virtuaalsele arvutile saab lisada tavalise pve webgui peal vIOMMU ning siis paigutatakse virtuaalse arvuti seadmed sh erinevad MLNX adapteri füüsilised pordid erinevatesse IOMMU gruppidesse |
||
| + | |||
| + | ====Tarkvara paigaldamine==== |
||
TODO |
TODO |
||
| 111. rida: | 126. rida: | ||
</pre> |
</pre> |
||
| + | ====Firmware uuendamine==== |
||
| − | Ning |
||
<pre> |
<pre> |
||
Redaktsioon: 30. juuni 2024, kell 01:02
Sissejuhatus
Mellanox riistvara
Väited
- Mellanox nö kangemad võrguseadmed jaotatakse kaheks suuremaks osakonnaks: 1. SmartNIC, 2. SuperNIC
- SmartNIC - nvidia connectx seadmed
- SuperNIC - nvidia bluefield seadmed
ConnectX seadmed
- 'connectx-6 lx' ja 'connectx-6 dx' seadmed on kõik ethernet seadmed (st mitte infiniband)
- 'connectx-6' seade on füüsiliselt universaalne ethernet/infiniband seade, st võimalik on tarkvaraliselt kaardi poole pöördudes lülitada ta käima ethernet või infiniband režiimis
Mellanox integratsioonid
dpdk
- https://enterprise-support.nvidia.com/s/article/mellanox-dpdk
- https://doc.dpdk.org/guides/nics/mlx5.html
kernel tls
TODO
inbox draiver ja utiliidid
Misc
# lspci | grep 3d:00 3d:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] 3d:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
devlink show andmed
# devlink dev show pci/0000:3d:00.0 pci/0000:3d:00.1
ja devlink info
# devlink dev info
pci/0000:3d:00.0:
driver mlx5_core
versions:
fixed:
fw.psid SM_1281000001000
running:
fw.version 26.35.2000
fw 26.35.2000
stored:
fw.version 26.35.2000
fw 26.35.2000
pci/0000:3d:00.1:
driver mlx5_core
versions:
fixed:
fw.psid SM_1281000001000
running:
fw.version 26.35.2000
fw 26.35.2000
stored:
fw.version 26.35.2000
fw 26.35.2000
ethtool andmed
# ethtool -i ens7f0np0 driver: mlx5_core version: 5.15.0-92-generic firmware-version: 26.35.2000 (SM_1281000001000) expansion-rom-version: bus-info: 0000:3d:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes
kus
- ethtool ja devlink-dev-info väljundis klapib kaardil oleva firmware versioon - 26.35.2000
MLNX EN tarkvara kasutamine
Mõisted
- MFT - NVIDIA Firmware Tools, tõenäoliselt algupäraselt Mellanox Firmware Tools
Kasutamine füüsilise seadme tervikuna passthru režiimis
Väited
- üldiselt proxmox v. 8 keskkonnas saab mellanox seadme anda üle virtuaalsele arvuti tavalisel viisil (valides pve webgui liidses 'Add -> PCI device' ja näidates esimese MLNX seadme; teine lisatakse automaatselt
- tundub, et füüsilist mellanox seadet ei saa tervikuna nö täiuslikult virtuaalsele arvutile edasi anda, ühe asjana puudub jääb sr-iov võimekus
- virtuaalses arvutis saab võrguseadet kasutada tema PF osas, VF ei ole ligipääsetav
- virtuaalsele arvutile saab lisada tavalise pve webgui peal vIOMMU ning siis paigutatakse virtuaalse arvuti seadmed sh erinevad MLNX adapteri füüsilised pordid erinevatesse IOMMU gruppidesse
Tarkvara paigaldamine
TODO
root@debian-mlnx-01:~# mount /root/mlnx-en-24.04-0.6.6.0-debian12.1-x86_64.iso /mnt/mlnx
root@debian-mlnx-01:~# find /lib/modules/6.1.0-22-amd64/ -type f -mmin -20 -ls | grep dkms 661960 4312 -rw-r--r-- 1 root root 4415205 Jun 29 22:07 /lib/modules/6.1.0-22-amd64/updates/dkms/mlx5_core.ko 661959 28 -rw-r--r-- 1 root root 25237 Jun 29 22:07 /lib/modules/6.1.0-22-amd64/updates/dkms/mlx_compat.ko 661961 8 -rw-r--r-- 1 root root 5565 Jun 29 22:07 /lib/modules/6.1.0-22-amd64/updates/dkms/mlx5_ib.ko 661962 52 -rw-r--r-- 1 root root 49445 Jun 29 22:07 /lib/modules/6.1.0-22-amd64/updates/dkms/mlxfw.ko 661963 208 -rw-r--r-- 1 root root 210797 Jun 29 22:07 /lib/modules/6.1.0-22-amd64/updates/dkms/mlxdevm.ko
Firmware uuendamine
root@debian-mlnx-01:~# apt-get install mlnx-fw-updater
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
linux-image-6.1.0-15-amd64
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
mlnx-fw-updater
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/50.2 MB of archives.
After this operation, 87.9 MB of additional disk space will be used.
Get:1 file:/mnt/mlnx/DEBS_ETH ./ mlnx-fw-updater 24.04-0.6.6.0 [50.2 MB]
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 63847 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_24.04-0.6.6.0_amd64.deb ...
Unpacking mlnx-fw-updater (24.04-0.6.6.0) ...
Setting up mlnx-fw-updater (24.04-0.6.6.0) ...
Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX6LX
Part Number: MCX631102AN-ADA_Ax
Description: ConnectX-6 Lx EN adapter card; 25GbE ; Dual-port SFP28; PCIe 4.0 x8; No Crypto
PSID: MT_0000000531
PCI Device Name: 01:00.0
Base GUID: e8ebd303000b7874
Base MAC: e8ebd30b7874
Versions: Current Available
FW 26.32.2004 26.41.1000
PXE 3.6.0502 3.7.0400
UEFI 14.25.0018 14.34.0012
Status: Update required
---------
Found 1 device(s) requiring firmware update...
Device #1: Updating FW ...
FSMST_INITIALIZE - OK
Writing Boot image component - OK
Done
Restart needed for updates to take effect.
Log File: /tmp/oaFVUkaJsl
Real log file: /tmp/mlnx_fw_update.log
root@debian-mlnx-01:~# less /tmp/mlnx_fw_update.log
CMD: mlxup -u --log-on-update --ssl-certificate /tmp/OloIGrYWuz/mlxfwmanager_sriov_dis_x86_64_4127-dir/ca-bundle.crt --current-dir /opt/mellanox/mlnx-fw-updater/ -L /tmp/oaFVUkaJsl -y -d 01:00.0
Querying Mellanox devices firmware ...
Device #1:
----------
...
Paistab, et tulemusena on kaardil olemas kaks versiooni firmwarest
root@debian-mlnx-01:~# devlink dev info
pci/0000:01:00.0:
driver mlx5_core
versions:
fixed:
fw.psid MT_0000000531
running:
fw.version 26.32.2004
fw 26.32.2004
stored:
fw.version 26.41.1000
fw 26.41.1000
pci/0000:01:00.1:
driver mlx5_core
versions:
fixed:
fw.psid MT_0000000531
running:
fw.version 26.32.2004
fw 26.32.2004
stored:
fw.version 26.41.1000
fw 26.41.1000
kus
- running version - 26.32.2004
- stored version - 26.41.1000
systemd unit mlnx-en.d
Tundub, et mlx driveritega tegeleb systemd unit
root@debian-mlnx-01:~# dpkg -S /etc/mlnx-en.conf
mlnx-en-utils: /etc/mlnx-en.conf
root@debian-mlnx-01:~# cat /etc/mlnx-en.conf
# Allow calling the service script with the option 'stop' for unloading the driver stack.
# This flag should be disabled when the OS root file system is on remote storage.
ALLOW_STOP=yes
# Run sysctl performance tuning script
RUN_SYSCTL=no
# Run /usr/sbin/mlnx_tune
RUN_MLNX_TUNE=no
# Load MLX4 modules
MLX4_LOAD=no
# Load MLX5 modules
MLX5_LOAD=yes
root@debian-mlnx-01:~# systemctl start mlnx-en.d
root@debian-mlnx-01:~# systemctl status mlnx-en.d
● mlnx-en.d.service - mlnx-en.d - configure Mellanox devices
Loaded: loaded (/lib/systemd/system/mlnx-en.d.service; enabled; preset: enabled)
Active: active (exited) since Sun 2024-06-30 00:49:29 EEST; 6s ago
Docs: file:/etc/mlnx-en.conf
Process: 1505 ExecStart=/etc/init.d/mlnx-en.d start (code=exited, status=0/SUCCESS)
Main PID: 1505 (code=exited, status=0/SUCCESS)
CPU: 385ms
Jun 30 00:49:26 debian-mlnx-01 systemd[1]: Starting mlnx-en.d.service - mlnx-en.d - configure Mellanox devices...
Jun 30 00:49:28 debian-mlnx-01 mlnx-en.d[1505]: [32B blob data]
Jun 30 00:49:29 debian-mlnx-01 systemd[1]: Finished mlnx-en.d.service - mlnx-en.d - configure Mellanox devices.
samal ajal dmesg väljundis
# dmesg -T -w [Sun Jun 30 00:49:26 2024] Compat-mlnx-ofed backport release: 7037b8d [Sun Jun 30 00:49:26 2024] Backport based on https://:@git-nbu.nvidia.com/r/a/mlnx_ofed/mlnx-ofa_kernel-4.0.git 7037b8d [Sun Jun 30 00:49:26 2024] compat.git: https://:@git-nbu.nvidia.com/r/a/mlnx_ofed/mlnx-ofa_kernel-4.0.git [Sun Jun 30 00:49:26 2024] mlx5_core 0000:01:00.0: firmware version: 26.32.2004 [Sun Jun 30 00:49:26 2024] mlx5_core 0000:01:00.0: 126.024 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x8 link) [Sun Jun 30 00:49:26 2024] mlx5_core 0000:01:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 24414Mbps [Sun Jun 30 00:49:26 2024] mlx5_core 0000:01:00.0: E-Switch: Total vports 2, per vport: max uc(128) max mc(2048) [Sun Jun 30 00:49:26 2024] mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged [Sun Jun 30 00:49:26 2024] mlx5_core 0000:01:00.0: mlx5_pcie_event:304:(pid 1398): PCIe slot advertised sufficient power (75W). [Sun Jun 30 00:49:26 2024] mlx5_core 0000:01:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 basic) [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.0 enp1s0f0np0: renamed from eth0 [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.1: firmware version: 26.32.2004 [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.1: 126.024 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x8 link) [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 24414Mbps [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.1: E-Switch: Total vports 2, per vport: max uc(128) max mc(2048) [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.1: Port module event: module 1, Cable plugged [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.1: mlx5_pcie_event:304:(pid 1391): PCIe slot advertised sufficient power (75W). [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 basic) [Sun Jun 30 00:49:27 2024] mlx5_core 0000:01:00.1 enp1s0f1np1: renamed from eth0
Kusjuures 'systemctl stop mlnx-en.d' eemaldab mlx moodulid mälust
root@debian-mlnx-01:~# lsmod | grep mlx mlx5_core 2269184 0 mlxfw 36864 1 mlx5_core mlxdevm 180224 1 mlx5_core mlx_compat 20480 2 mlxdevm,mlx5_core psample 20480 1 mlx5_core tls 135168 1 mlx5_core pci_hyperv_intf 16384 1 mlx5_core root@debian-mlnx-01:~# systemctl stop mlnx-en.d root@debian-mlnx-01:~# lsmod | grep mlx root@debian-mlnx-01:~#
Misc
- https://www.youtube.com/watch?v=XLPgDEbUMgk - 'How to set Mellanox ConnectX VPI to Ethernet or Infiniband in Linux'
