ISCSI offload
Sissejuhatus
TODO
Tööpõhimõte
TODO
Riistvara
Et toeta üldse, st nö pole lootust ka
TODO
Võrgukaardi mudel põhimõttelist toetab, mis ei tähenda, et praktiliselt kasutada saab (nt funktsionaalsust sisselülitav riistvaraline võti on puudu, punni kujuline)
# lspci | grep -i ether 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12) 07:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12) 16:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12) 18:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)
Toetab põhimõtteliselt, aga midagi siiski puudub
[17763.496402] bnx2i: LOM is not enable to offload iSCSI connections, dev=eth0
Redhat v. 7
iSCSI initiatori tarkvara paigaldamiseks sobib öelda
# yum install iscsi-initiator-utils # yum install iscsi-initiator-utils-iscsiuio
Tulemusena tekivad failisüsteemi
- /etc/iscsi/iscsid.conf - peamine seadistusfail
- /usr/sbin/iscsid - Open-iSCSI daemon
- /usr/sbin/iscsiadm - open-iscsi administration utility
- /usr/sbin/iscsiuio - iSCSI UserSpace I/O driver
Offload jt liideste nimekirja esitamine
# iscsiadm -m iface default tcp,<empty>,<empty>,<empty>,<empty> iser iser,<empty>,<empty>,<empty>,<empty> bnx2i.78:e7:d1:f6:07:77 bnx2i,78:e7:d1:f6:07:77,<empty>,<empty>,<empty> bnx2i.78:e7:d1:f6:07:75 bnx2i,78:e7:d1:f6:07:75,<empty>,<empty>,<empty>
Offload liidesele ip aadressi omistamine (tavalisel ethernet liidesel on reeglina mac ühe võrra väiksema viimase numbriga ning ip aadress seadistatud sarnaselt ühe võrra väiksem)
# iscsiadm -m iface -I bnx2i.78:e7:d1:f6:07:75 -o update -n iface.ipaddress -v 10.0.0.211
Discovery üle offload liidese
# iscsiadm -m discovery -t st -p 10.0.0.209:3260 -I bnx2i.78:e7:d1:f6:07:75 -P 1 Target: iqn.201410.imool.ee.test1:test1 Portal: 10.0.0.209:3260,1 Iface Name: bnx2i.78:e7:d1:f6:07:75
Külgeühendumiseks sobib öelda
# iscsiadm -m node -T iqn.201410.imool.ee.test1:test1 -p 10.0.0.209:3260 -l Logging in to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.0.209,3260] (multiple) Login to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.0.209,3260] successful.
Töötavale sesioonile vastab selline väljund
# iscsiadm -m session -P 1 Target: iqn.201410.imool.ee.test1:test1 (non-flash) Current Portal: 10.0.0.209:48140,1 Persistent Portal: 10.0.0.209:3260,1 ********** Interface: ********** Iface Name: bnx2i.78:e7:d1:f6:07:75 Iface Transport: bnx2i Iface Initiatorname: iqn.1994-05.com.redhat:7fe6981fef65 Iface IPaddress: 10.0.0.211 Iface HWaddress: 78:e7:d1:f6:07:75 Iface Netdev: enp2s0f0 SID: 3 iSCSI Connection State: LOGGED IN iSCSI Session State: LOGGED_IN Internal iscsid Session State: NO CHANGE
Kui selliselt tekkinud plokkseadet kasutada ning võrku kõrvalt kuulata targetis, siis on näha, et pöördutakse .211 st offload komponendile omistatud ip aadressilt.
Töötavad protsessid
# ps aux | grep iscsi .. root 2736 0.0 0.0 13832 512 ? Ss 01:41 0:00 /usr/sbin/iscsid root 2737 0.0 0.1 16480 4220 ? S<Ls 01:41 0:00 /usr/sbin/iscsid root 2740 5.7 1.0 248492 43640 ? S<Lsl 01:41 2:15 /usr/sbin/iscsiuio ...
dmesg väljundis paistab taustal midagi sellist
[ 1071.887882] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.19 (December 19, 2013) [ 1071.891303] bnx2 0000:02:00.0 enp2s0f0: Added CNIC device [ 1071.895742] bnx2 0000:02:00.1 enp2s0f1: Added CNIC device [ 1071.905159] Loading iSCSI transport class v2.0-870. [ 1071.913716] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.6.2 (Jun 06, 2013) [ 1071.917250] iscsi: registered transport (bnx2i) [ 1071.920896] scsi3 : Broadcom Offload iSCSI Initiator [ 1071.924833] bnx2i [02:00.01]: ISCSI_INIT passed [ 1071.928984] scsi4 : Broadcom Offload iSCSI Initiator [ 1071.932802] bnx2i [02:00.00]: ISCSI_INIT passed [ 1249.378334] connection1:0: bnx2i: conn update - MBL 0xfffc00 FBL 0x40000MRDSL_I 0x2000 MRDSL_T 0x2000 [ 1423.291754] connection2:0: bnx2i: conn update - MBL 0x40000 FBL 0x10000MRDSL_I 0x40000 MRDSL_T 0x2000
Asjassepuutuvad failid
# find /var/lib/iscsi -type f -ls 9496443 4 -rw------- 1 root root 474 Feb 10 01:38 /var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:77 9496444 4 -rw------- 1 root root 506 Feb 10 01:40 /var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:75 4196315 4 -rw------- 1 root root 2127 Feb 10 02:18 /var/lib/iscsi/nodes/iqn.201410.piia.ee.test1:test1/10.0.0.209,3260,1/bnx2i.78:e7:d1:f6:07:75 1088931 4 -rw------- 1 root root 552 Feb 10 02:18 /var/lib/iscsi/send_targets/10.0.0.209,3260/st_config
kus
# cat '/var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:75' # BEGIN RECORD 6.2.0.873-21 iface.iscsi_ifacename = bnx2i.78:e7:d1:f6:07:75 iface.ipaddress = 10.0.0.211 iface.hwaddress = 78:e7:d1:f6:07:75 iface.transport_name = bnx2i iface.vlan_id = 0 iface.vlan_priority = 0 iface.iface_num = 0 iface.mtu = 0 iface.port = 0 iface.tos = 0 iface.ttl = 0 iface.tcp_wsf = 0 iface.tcp_timer_scale = 0 iface.def_task_mgmt_timeout = 0 iface.erl = 0 iface.max_receive_data_len = 0 iface.first_burst_len = 0 iface.max_outstanding_r2t = 0 iface.max_burst_len = 0 # END RECORD
Kasulikud lisamaterjalid
SupportProduct -> Documentation -> Red Hat Enterprise Linux 7 -> Storage Administration Guide -> 26.12. CONFIGURING ISCSI OFFLOAD AND INTERFACE BINDING
Ubuntu v. 14.04
TODO
Debian v. 8 Jessie
Riistvara, BIOS
Broadcom NetXtreme II Ethernet Boot Agent v5.0.5 <F9 = Setup> Copyright (C) 2000-2009 Broadcom Corporation All rights reserved. Press Ctrl-S to Configure Device (MAC Address - 78E7D1F60774)
Seadmed
# lspci | grep -i ether 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
bnx2i draiveri laadimisel öeldakse dmesgi
[ 117.382139] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.20 (March 14, 2014) [ 117.382275] bnx2 0000:02:00.0 eth0: Added CNIC device [ 117.383534] bnx2 0000:02:00.1 eth1: Added CNIC device [ 117.397584] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.6.2 (Jun 06, 2013) [ 117.397726] iscsi: registered transport (bnx2i) [ 117.401852] scsi3 : Broadcom Offload iSCSI Initiator [ 117.405854] scsi4 : Broadcom Offload iSCSI Initiator
# iscsiadm -m iface default tcp,<empty>,<empty>,<empty>,<empty> iser iser,<empty>,<empty>,<empty>,<empty> bnx2i.00:00:00:00:00:00 bnx2i,00:00:00:00:00:00,<empty>,<empty>,<empty> bnx2i.10:e7:d1:f6:07:75 bnx2i,10:e7:d1:f6:07:75,10.0.1.211,<empty>,<empty>
ip aadressi omistamine
# iscsiadm -m iface -I bnx2i.00:e7:d1:f6:07:75 -o update -n iface.ipaddress -v 10.0.1.209
tulemusena on
# cat /etc/iscsi/ifaces/bnx2i.10\:e7\:d1\:f6\:07\:75 # BEGIN RECORD 2.0-873 iface.iscsi_ifacename = bnx2i.10:e7:d1:f6:07:75 iface.ipaddress = 10.0.1.211 iface.hwaddress = 10:e7:d1:f6:07:75 iface.transport_name = bnx2i iface.vlan_id = 0 iface.vlan_priority = 0 iface.iface_num = 0 iface.mtu = 0 iface.port = 0
Discovery tegemiseks
# /usr/bin/iscsiadm -I bnx2i.00:e7:d1:f6:07:75 -m discovery -t sendtargets -p 10.0.1.209 iscsiadm: can not connect to iSCSI daemon (111)! iscsiadm: uIP daemon is not up iscsiadm: Could not set host net params (err 20)
Debian v.6 Squeeze
Discovery bnx2i kaudu
# /usr/bin/iscsiadm -I bnx2i.78:e7:d1:f6:07:75 -m discovery -t sendtargets -p 10.0.1.209 10.0.1.209:3260,1 iqn.201410.imool.ee.test1:test1
Login
# iscsiadm -m node -I bnx2i.78:e7:d1:f6:07:75 -T iqn.201410.imool.ee.test1:test1 -p 10.0.1.209:3260 -l Logging in to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.1.209,3260] iscsiadm: Could not login to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.1.209,3260]: iscsiadm: initiator reported error (4 - encountered connection failure)
FCoE
Kasulikud lisamaterjalid
- http://en.community.dell.com/techcenter/b/techcenter/archive/2012/07/24/red-hat-enterprise-linux-6-3-on-dell-poweredge-servers-features-and-native-drivers
- http://www.qlogic.com/OEMPartnerships/Dell/Documents/Converged/DataSheet_57810SDell_CNA_DualPort_10GBASE-T.pdf
- http://forums.qlogic.com/SupportCenter/Customer_support_main?id=kA28000000008bwCAA
- http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c04448818-3.pdf
LIO target
TODO
Tundub, et 2018 sügisel on LIO (Linux-IO) kõige õigem lahendus iSCSI protokolli abil plokkseadme ressurssi üle nö tavalise ethernet võrgu kasutamiseks. Iseenesest LIO võimaldab teha muudki, LIO sisaldab
- erinevate back storage toetust (fail, partitsioon, lvm jt )
- erinevate andmevahetuse kanalite toetust (ethernet, fc jt)
Tööpõhimõte
LIO töötamisega ja sealhulgas võrgus nt iscsi teenuse pakkumisega ei kaasneb ühtegi user-space protsessi; tööd teeb kernel ja kerneli moodulid. iSCSI lahenduse määratlevad sellised objektid ja nende vahelised seosed
- plokkseade
- target portal
- lun
- acl
- initiatior (klient)
LIO target seadistatakse sysfs failisüsteemis /sys/kernel/config/target alla jääva osaga töötades. Muudatuste haldamist lihtsustab targetcli utiliit.
Tarkvara paigaldamine
2018 sügisel sobib Debian v. 9 keskkonnas öelda
# apt-get install targetcli-fb
CentOS v. 7 keskkonnas sobib öelda
# yum install targetcli
Seadistamine
Ettevalmistused
- vajalik on teada iscsi initiator arvuti iqn väärtust (olgu see nt iqn.1993-08.org.debian:01:aaaaaaaaaaa)
# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1993-08.org.debian:01:aaaaaaaaaaa
- antud näites ei piirata ligipääsu iscsi ressursile parooliga, ainult kliendi iqn väärtusega (täindava ligipääsu saab seadistada nt paketifiltriga)
Käsuga
# targetcli
jõuab sysfs failisüsteemi haldamisse interaktiivsesse režiimi.
- plokkseadme kirjeldamine
/backstores/block create iscsi_block_1 /dev/vdb
- target iqn kirjeldamine ja portal kirjeldamine
/iscsi create iqn.2018-12.moraal.srv:storage.target03
- iqn kirjeldamine, millelt on lubatud ligi pääseda
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls create iqn.1993-08.org.debian:01:aaaaaaaaaaa
- lun tekitamine ja sidumine targetiga
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/luns create /backstores/block/iscsi_block_1
Paistab, et peale ülaltoodud käskude ütlemist muudatused kehtestatakse. Seadistused salvesatakse faili
/etc/rtslib-fb-target/saveconfig.json
Misc
- iqn eemaldamine, millelt on lubatud ligi pääseda
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls delete iqn.1993-08.org.debian:01:aaaaaaaaaaa
- portal eemaldamine (praktiliselt lõpetatakse ka 0.0.0.0:3260/tcp kuulamine)
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls delete 0.0.0.0 3260
- target ja block seade eemaldamine
/iscsi> /iscsi delete iqn.2021-01.tartu:storage.target01 /backstores> /backstores/fileio delete disc2_sda
Haldamine
root@cifs:~# targetcli sessions alias: pm-moraal-4-v52 sid: 1 type: Normal session-state: LOGGED_IN
LIO teenust käivitab
# systemctl status rtslib-fb-targetctl
LIO'ga tegelevad kerneli moodulid
# lsmod | grep ^target target_core_user 20480 0 target_core_pscsi 20480 0 target_core_file 20480 2 target_core_iblock 20480 2 ...
Võrgus teenindab kernel-space
# netstat -lnpt | grep -- - Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN -
Tundub, et Proxmox peal saab kergesti hakata iscsi ressursse kaustama, aga kaustamist ära lõpetada polegi webgui kaudu võimalik, käsurealt sobib öelda
# iscsiadm -m session -P 1 # iscsiadm -m node -T iqn.2018-12.moraal.srv:storage.target02 -p 192.168.10.127 -u # iscsiadm -m node -T iqn.2018-12.moraal.srv:storage.target02 -p 192.168.10.127:3260 -o delete
Proxmox pealt kasutamiseks võib olla tark esmalt host peal /etc/lvm/lvm.conf peale seadistada filter ja global_filter
TODO
Kasulikud lisamaterjalid
- http://www.linux-iscsi.org/Doc/LIO%20Admin%20Manual.pdf
- https://wiki.rvijay.in/index.php/Configuring_iSCSI_target_using_'targetcli'
- http://linux-iscsi.org/wiki/ISCSI
- https://www.server-world.info/en/note?os=Debian_9&p=iscsi&f=1
- https://www.thegeekdiary.com/how-to-configure-iscsi-target-and-initiator-using-targetcli-in-centos-rhel-7/
- https://www.linuxhelp.com/how-to-create-and-setup-luns-using-lvm
2024 kevad täiendused - iSER (iSCSI Extensions for RDMA)
Kasutada on üks füüsiline kahe pordiga võrgukaart
# lspci | grep -i net 81:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02) 81:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)
Operatsioonisüsteemi tavalised võrguseadmed on sellised
root@pve-02:~# ethtool enp129s0f1np1 Settings for enp129s0f1np1: Supported ports: [ FIBRE ] Supported link modes: 1000baseT/Full 25000baseCR/Full 25000baseSR/Full 1000baseX/Full 10000baseCR/Full 10000baseSR/Full 10000baseLR/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: None RS BASER Advertised link modes: 25000baseCR/Full 10000baseCR/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Advertised FEC modes: None RS BASER Link partner advertised link modes: Not reported Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 25000Mb/s Duplex: Full Auto-negotiation: on Port: Direct Attach Copper PHYAD: 0 Transceiver: internal Supports Wake-on: g Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes root@pve-02:~# ethtool -i enp129s0f1np1 driver: ice version: 6.5.13-5-pve firmware-version: 4.40 0x8001c7d4 1.3534.0 expansion-rom-version: bus-info: 0000:81:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Oluline on, et see ip konf millega opereeritakse järgnevalt oleks kinnitatud füüsilise seadme külge (st mitte nii, et arvutis on kasutusel nt OVS virtual switch ning ip on vlan47 küljes, sama ovs bridge küljes on füüsiline enp129s0f1np1 jne, st siis ei toimi)
root@pve-02:~# ip addr show dev enp129s0f1np1: 3: enp129s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 3c:ec:ef:e6:69:b9 brd ff:ff:ff:ff:ff:ff inet 10.47.218.226/24 scope global enp129s0f1np1 valid_lft forever preferred_lft forever inet6 fe80::3eec:efff:fee6:69b9/64 scope link valid_lft forever preferred_lft forever
Paistab devlink vaatest sedasi
# devlink dev info pci/0000:81:00.0: driver ice serial_number 00-01-00-ff-ff-00-00-00 versions: fixed: board.id K58132-000 running: fw.mgmt 7.4.13 fw.mgmt.api 1.7.11 fw.mgmt.build 0xded4446f fw.undi 1.3534.0 fw.psid.api 4.40 fw.bundle_id 0x8001c7d4 fw.app.name ICE OS Default Package fw.app 1.3.36.0 fw.app.bundle_id 0xc0000001 fw.netlist 4.4.5000-2.15.0 fw.netlist.build 0x0ba411b9 stored: fw.undi 1.3534.0 fw.psid.api 4.40 fw.bundle_id 0x8001c7d4 fw.netlist 4.4.5000-2.15.0 fw.netlist.build 0x0ba411b9 ...
Muu hulgas toetab see seade iwarp ja rocev2 protokolle, devlink vaatest paistab see nii
# devlink dev param pci/0000:81:00.0: name enable_roce type generic values: cmode runtime value true name enable_iwarp type generic values: cmode runtime value false pci/0000:81:00.1: name enable_roce type generic values: cmode runtime value false name enable_iwarp type generic values: cmode runtime value true
Korraga saab olla aktiivne üks või teine, nende vahel valmine toimub nt selliselt devlink abil
# devlink dev param set pci/0000:81:00.1 name enable_iwarp value false cmode runtime
rocev2 kasutamiseks on asjakohen süsteemidesse paigaldada omajagu rdma ja infiniband traditsiooniga tarkvara, nt
apt-get install rdma-core apt-get install perftest apt-get install ibverbs-utils apt-get install infiniband-diags
Asjasse puutuvad sellised driverid
- ice - põhi draiver
- irdma - intel võrgukaartide (2024 aastal e810 ja midagi vanemat ka) - 'intel rdma'
Seadmed kui rdma seadmed paistavad
root@pve-01:~# rdma link link rocep129s0f0/1 state ACTIVE physical_state LINK_UP netdev enp129s0f0np0 link rocep129s0f1/1 state ACTIVE physical_state LINK_UP netdev enp129s0f1np1
Kahe otse kaabliga süsteemi uurimiseks sobib öelda ühes ja teises arvutis nt nii, pve-02 on nö rping server ja pve-01 on rping klient; seejuures on iseloomulik, et tavalise võrguliidese peal tavalisel viisil võrku pealt kuulates ei ole midagi näha
root@pve-02:~# rping -s -a 10.47.218.226 -v server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10 root@pve-01:~# rping -c -a 10.47.218.226 -v -C 5 ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv client DISCONNECT EVENT...
Veel huvitavaid utiliite, seejuures on esimene seade 'devlink dev param ...' abil seadistatud rocev2 režiimi ja teine iwarp režiimi
# ibv_devices device node GUID ------ ---------------- rocep129s0f0 3eeceffffee667b2 iwp129s0f1 3eeceffffee667b3 # ibv_devinfo hca_id: rocep129s0f0 transport: InfiniBand (0) fw_ver: 1.71 node_guid: 3eec:efff:fee6:67b2 sys_image_guid: 3eec:efff:fee6:67b2 vendor_id: 0x8086 vendor_part_id: 5531 hw_ver: 0x2 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 1 port_lmc: 0x00 link_layer: Ethernet hca_id: iwp129s0f1 transport: iWARP (1) fw_ver: 1.71 node_guid: 3eec:efff:fee6:67b3 sys_image_guid: 3eec:efff:fee6:67b3 vendor_id: 0x8086 vendor_part_id: 5531 hw_ver: 0x2 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 1 port_lmc: 0x00 link_layer: Ethernet
Töötaval juhul roce seadmega
root@pve-02:~# ib_send_bw -i 1 -d rocep129s0f1 ************************************ * Waiting for client to connect... * ************************************ --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF Device : rocep129s0f1 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : OFF RX depth : 512 CQ Moderation : 1 Mtu : 1024[B] Link type : Ethernet GID index : 1 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0x01 QPN 0x0004 PSN 0x807cd5 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:47:218:226 remote address: LID 0x01 QPN 0x0004 PSN 0xba4fee GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:47:218:225 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] Conflicting CPU frequency values detected: 3696.643000 != 3000.000000. CPU Frequency is not max. 65536 1000 0.00 2762.05 0.044193 ---------------------------------------------------------------------------------------
ja esimene arvuti kliendina
root@pve-01:~# ib_send_bw -i 1 10.47.218.226 -d rocep129s0f1 --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF Device : rocep129s0f1 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : OFF TX depth : 128 CQ Moderation : 1 Mtu : 1024[B] Link type : Ethernet GID index : 1 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0x01 QPN 0x0004 PSN 0xba4fee GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:47:218:225 remote address: LID 0x01 QPN 0x0004 PSN 0x807cd5 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:47:218:226 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] Conflicting CPU frequency values detected: 3698.021000 != 3000.000000. CPU Frequency is not max. 65536 1000 2757.88 2757.84 0.044126 ---------------------------------------------------------------------------------------
kus
- tundub, et ib_send_bw on infiniband põlvnevusega utiliit
- Transport type on IB ehk infiniband, st nii nagu roce protokolli tööpõhimõte on
- Link type on ethernet
Mittetöötav juhum - kuna kasutatakse iwarp seadmel
root@pve-02:~# ib_send_bw -i 1 -d iwp129s0f1 ************************************ * Waiting for client to connect... * ************************************ --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF Device : iwp129s0f1 Number of qps : 1 Transport type : IW Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : OFF RX depth : 512 CQ Moderation : 1 Mtu : 1024[B] Link type : Ethernet GID index : 0 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- ethernet_read_keys: Couldn't read remote address Unable to read to socket/rdma_cm Failed to exchange data between server and clients Failed to deallocate PD - Device or resource busy Failed to destroy resources
Mittetöötav juhtum - kuna seade on küll arvutis olemas, aga seal ei ole kinnitatud kõnealust ip aadressi
root@pve-02:~# ib_send_bw -i 1 -d rocep129s0f0 ************************************ * Waiting for client to connect... * ************************************ --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF Device : rocep129s0f0 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : OFF RX depth : 512 CQ Moderation : 1 Mtu : 1024[B] Link type : Ethernet GID index : 0 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- Failed to modify QP 5 to RTR Unable to Connect the HCA's through the link
Kõige praktilisem kasutusjuht on iscsi kasutamine iser + rocev2 viisil. Tulemusena on andmevahetuse kiirus 2x suurem võrreldes nö tavalisega. Target näeb välja selline
/> ls / o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- iscsi_block_md127 ............................................................. [/dev/md127 (27.9TiB) write-thru activated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2003-01.org.setup.lun.test .................................................................................... [TPGs: 1] | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.1993-08.org.debian:01:b65e1ba35869 ................................................... [1-way auth, Mapped LUNs: 1] | | o- mapped_lun0 ..................................................................... [lun0 block/iscsi_block_md127 (rw)] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 ........................................................ [block/iscsi_block_md127 (/dev/md127) (default_tg_pt_gp)] | o- portals .................................................................................................... [Portals: 1] | o- 10.47.218.226:3261 ............................................................................................. [iser] o- loopback ......................................................................................................... [Targets: 0] o- srpt ............................................................................................................. [Targets: 0] o- vhost ............................................................................................................ [Targets: 0] o- xen-pvscsi ....................................................................................................... [Targets: 0]
ning initiator
root@pve-01:~# iscsiadm -m discovery -t st -p 10.47.218.226:3261 root@pve-01:~# iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -p 10.47.218.226:3261 -o update -n iface.transport_name -v iser root@pve-01:~# iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -p 10.47.218.226:3261 -l root@pve-01:~# lsscsi -s ... root@pve-01:~# iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -p 10.47.218.226:3261 -u root@pve-01:~# iscsiadm -m discoverydb -t sendtargets -p 10.47.218.226:3261 -o delete root@pve-01:~# iscsiadm -m discoverydb
Kasulikud lisamaterjalid
- 'Intel E800 devices now support iWARP and RoCE protocols' - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/8.7_release_notes/new-features
- https://forum.proxmox.com/threads/mellanox-connectx-4-lx-and-brigde-vlan-aware-on-proxmox-8-0-1.130902/
- https://docs.nvidia.com/networking/display/mftv4270/updating+firmware+using+ethtool/devlink+and+-mfa2+file
- https://edc.intel.com/content/www/ca/fr/design/products/ethernet/appnote-e810-eswitch-switchdev-mode-config-guide/eswitch-mode-switchdev-and-legacy/
- nvidia-mlnx-ofed-documentation-v23-07.pdf
- https://enterprise-support.nvidia.com/s/article/How-To-Enable-Verify-and-Troubleshoot-RDMA
- https://enterprise-support.nvidia.com/s/article/howto-configure-lio-enabled-with-iser-for-rhel7-inbox-driver
- https://dri.freedesktop.org/docs/drm/networking/devlink/ice.html
2024 kevad täiendused - VF
TODO
Kasulikud lisamaterjalid
- https://clouddocs.f5.com/cloud/public/v1/kvm/kvm_intel.html
- https://edc.intel.com/content/www/ca/fr/design/products/ethernet/appnote-e810-eswitch-switchdev-mode-config-guide/configuration-steps/
- https://markmcb.com/freebsd/vs_linux/sriov_is_first_class/
- https://serverfault.com/questions/931228/error-in-enbling-sr-iov-with-ubuntu-18-04-on-intel-ixgbe-intel-x550-dell-r64
- https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/pdf/configuring_and_managing_virtualization/red_hat_enterprise_linux-9-configuring_and_managing_virtualization-en-us.pdf
- https://docs.nvidia.com/networking/display/mlnxofedv23100550/kubernetes+using+sr-iov
- https://www.supermicro.com/support/faqs/faq.cfm?faq=31936
Kasulikud lisamaterjalid
- http://h10032.www1.hp.com/ctg/Manual/c01555907.pdf
- https://github.com/mikechristie/open-iscsi
- http://www.dell.com/downloads/global/power/ps4q08-20080208-Broadcom.pdf
- http://www.broadcom.com/collateral/pb/5709C-PB02-R.pdf
- http://www.broadcom.com/collateral/pb/5719-PB01-R.pdf
- http://www.broadcom.com/collateral/pb/5720_Chip_PB.pdf
- http://forum.broadcom.com/showthread.php?91157-Broadcom-NetXtreme-II-1Gb-and-10Gb-Products-Now-Serviced-by-QLogic
- http://www.broadcom.com/support/ethernet_nic/downloaddrivers.php
- http://www.qlogic.com/solutions/Pages/Ethernet-NIC-FAQs.aspx