Mellanox ConnectX-6 Lx EN

Mellanox riistvara

Väited

• Mellanox nö kangemad võrguseadmed jaotatakse kaheks suuremaks osakonnaks: 1. SmartNIC, 2. SuperNIC
• SmartNIC - nvidia connectx seadmed
• SuperNIC - nvidia bluefield seadmed

ConnectX seadmed

• 'connectx-6 lx' ja 'connectx-6 dx' seadmed on kõik ethernet seadmed (st mitte infiniband)
• 'connectx-6' seade on füüsiliselt universaalne ethernet/infiniband seade, st võimalik on tarkvaraliselt kaardi poole pöördudes lülitada ta käima ethernet või infiniband režiimis

Mellanox integratsioonid

dpdk

• https://enterprise-support.nvidia.com/s/article/mellanox-dpdk
• https://doc.dpdk.org/guides/nics/mlx5.html

kernel tls

TODO

Misc

# lspci | grep 3d:00
3d:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
3d:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]

devlink show andmed

# devlink dev show
pci/0000:3d:00.0
pci/0000:3d:00.1

ja devlink info

# devlink dev info
pci/0000:3d:00.0:
  driver mlx5_core
  versions:
      fixed:
        fw.psid SM_1281000001000
      running:
        fw.version 26.35.2000
        fw 26.35.2000
      stored:
        fw.version 26.35.2000
        fw 26.35.2000
pci/0000:3d:00.1:
  driver mlx5_core
  versions:
      fixed:
        fw.psid SM_1281000001000
      running:
        fw.version 26.35.2000
        fw 26.35.2000
      stored:
        fw.version 26.35.2000
        fw 26.35.2000

ethtool andmed

# ethtool -i ens7f0np0
driver: mlx5_core
version: 5.15.0-92-generic
firmware-version: 26.35.2000 (SM_1281000001000)
expansion-rom-version: 
bus-info: 0000:3d:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

kus

• ethtool ja devlink-dev-info väljundis klapib kaardil oleva firmware versioon - 26.35.2000

MLNX EN tarkvara kasutamine

TODO

root@debian-mlnx-01:~# apt-get install mlnx-fw-updater
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  linux-image-6.1.0-15-amd64
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  mlnx-fw-updater
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/50.2 MB of archives.
After this operation, 87.9 MB of additional disk space will be used.
Get:1 file:/mnt/mlnx/DEBS_ETH ./ mlnx-fw-updater 24.04-0.6.6.0 [50.2 MB]
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 63847 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_24.04-0.6.6.0_amd64.deb ...
Unpacking mlnx-fw-updater (24.04-0.6.6.0) ...
Setting up mlnx-fw-updater (24.04-0.6.6.0) ...
Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX6LX
  Part Number:      MCX631102AN-ADA_Ax
  Description:      ConnectX-6 Lx EN adapter card; 25GbE ; Dual-port SFP28; PCIe 4.0 x8; No Crypto
  PSID:             MT_0000000531
  PCI Device Name:  01:00.0
  Base GUID:        e8ebd303000b7874
  Base MAC:         e8ebd30b7874
  Versions:         Current        Available     
     FW             26.32.2004     26.41.1000    
     PXE            3.6.0502       3.7.0400      
     UEFI           14.25.0018     14.34.0012    

  Status:           Update required

---------
Found 1 device(s) requiring firmware update...

Device #1: Updating FW ...     
FSMST_INITIALIZE -   OK          
Writing Boot image component -   OK          
Done

Restart needed for updates to take effect.
Log File: /tmp/oaFVUkaJsl
Real log file: /tmp/mlnx_fw_update.log

root@debian-mlnx-01:~# less /tmp/mlnx_fw_update.log
CMD: mlxup -u --log-on-update --ssl-certificate /tmp/OloIGrYWuz/mlxfwmanager_sriov_dis_x86_64_4127-dir/ca-bundle.crt --current-dir /opt/mellanox/mlnx-fw-updater/  -L /tmp/oaFVUkaJsl -y -d 01:00.0 
Querying Mellanox devices firmware ...

Device #1:
----------

...

Misc

• https://www.youtube.com/watch?v=XLPgDEbUMgk - 'How to set Mellanox ConnectX VPI to Ethernet or Infiniband in Linux'

Kasulikud lisamaterjalid

• https://network.nvidia.com/pdf/prod_software/Ubuntu_20_04_Inbox_Driver_User_Manual.pdf