ISCSI offload: erinevus redaktsioonide vahel
Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti
| (ei näidata sama kasutaja 4 vahepealset redaktsiooni) | |||
| 365. rida: | 365. rida: | ||
===2024 kevad täiendused - iSER (iSCSI Extensions for RDMA)=== |
===2024 kevad täiendused - iSER (iSCSI Extensions for RDMA)=== |
||
| + | Vt https://www.auul.pri.ee/wiki/Intel_E810 |
||
| + | ===Ubuntu 24.04 ja targetcli-fb=== |
||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | Oluline on, et see ip konf millega opereeritakse järgnevalt oleks kinnitatud füüsilise seadme külge (st mitte nii, et arvutis on kasutusel nt OVS virtual switch ning ip on vlan47 küljes, sama ovs bridge küljes on füüsiline enp129s0f1np1 jne, st siis ei toimi) |
||
<pre> |
<pre> |
||
| + | root@tgt-zfs-over-iscsi:~# apt-get install targetcli-fb |
||
| − | root@pve-02:~# ip addr show dev enp129s0f1np1: |
||
| − | 3: enp129s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 |
||
| − | link/ether 3c:ec:ef:e6:69:b9 brd ff:ff:ff:ff:ff:ff |
||
| − | inet 10.47.218.226/24 scope global enp129s0f1np1 |
||
| − | valid_lft forever preferred_lft forever |
||
| − | inet6 fe80::3eec:efff:fee6:69b9/64 scope link |
||
| − | valid_lft forever preferred_lft forever |
||
| − | </pre> |
||
| + | .. |
||
| + | Unpacking targetcli-fb (1:2.1.53-1ubuntu3) ... |
||
| + | Setting up python3-wcwidth (0.2.5+dfsg1-1.1ubuntu1) ... |
||
| + | Setting up python3-typing-extensions (4.10.0-1) ... |
||
| + | Setting up python3-pyudev (0.24.0-1) ... |
||
| + | Setting up python3-rtslib-fb (2.1.74-0ubuntu5) ... |
||
| + | Created symlink /etc/systemd/system/multi-user.target.wants/rtslib-fb-targetctl.service → /usr/lib/systemd/system/rtslib-fb-targetctl.service. |
||
| + | Setting up python3-urwid (2.6.10-1) ... |
||
| + | Setting up python3-configshell-fb (1:1.1.28-2.1) ... |
||
| + | /usr/lib/python3/dist-packages/configshell/console.py:39: SyntaxWarning: invalid escape sequence '\[' |
||
| + | _re_ansi_seq = re.compile('(\033\[..?m)') |
||
| + | /usr/lib/python3/dist-packages/configshell/shell.py:116: SyntaxWarning: invalid escape sequence '\+' |
||
| + | var = Word(alphanums + '?;&*$!#,=_\+/.<>()~@:-%[]') |
||
| + | /usr/lib/python3/dist-packages/configshell/shell.py:118: SyntaxWarning: invalid escape sequence '\-' |
||
| + | keyword = Word(alphanums + '_\-') |
||
| + | /usr/lib/python3/dist-packages/configshell/shell.py:124: SyntaxWarning: invalid escape sequence '\[' |
||
| + | pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \ |
||
| + | /usr/lib/python3/dist-packages/configshell/shell.py:124: SyntaxWarning: invalid escape sequence '\[' |
||
| + | pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \ |
||
| + | /usr/lib/python3/dist-packages/configshell/shell.py:131: SyntaxWarning: invalid escape sequence '\|' |
||
| + | readline.set_completer_delims('\t\n ~!#$^&(){}\|;\'",?') |
||
| + | /usr/lib/python3/dist-packages/configshell_fb/console.py:39: SyntaxWarning: invalid escape sequence '\[' |
||
| + | _re_ansi_seq = re.compile('(\033\[..?m)') |
||
| + | /usr/lib/python3/dist-packages/configshell_fb/shell.py:116: SyntaxWarning: invalid escape sequence '\+' |
||
| + | var = Word(alphanums + '?;&*$!#,=_\+/.<>()~@:-%[]') |
||
| + | /usr/lib/python3/dist-packages/configshell_fb/shell.py:118: SyntaxWarning: invalid escape sequence '\-' |
||
| + | keyword = Word(alphanums + '_\-') |
||
| + | /usr/lib/python3/dist-packages/configshell_fb/shell.py:124: SyntaxWarning: invalid escape sequence '\[' |
||
| + | pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \ |
||
| + | /usr/lib/python3/dist-packages/configshell_fb/shell.py:124: SyntaxWarning: invalid escape sequence '\[' |
||
| + | pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \ |
||
| + | /usr/lib/python3/dist-packages/configshell_fb/shell.py:131: SyntaxWarning: invalid escape sequence '\|' |
||
| + | readline.set_completer_delims('\t\n ~!#$^&(){}\|;\'",?') |
||
| + | Setting up targetcli-fb (1:2.1.53-1ubuntu3) ... |
||
| + | /usr/lib/python3/dist-packages/targetcli/ui_root.py:161: SyntaxWarning: invalid escape sequence '\s' |
||
| + | backups = [line for line in prefs.read().splitlines() if re.match('^max_backup_files\s*=', line)] |
||
| + | Processing triggers for man-db (2.12.0-4build2) ... |
||
| + | Scanning processes... |
||
| + | Scanning candidates... |
||
| + | Scanning linux images... |
||
| + | Running kernel seems to be up-to-date. |
||
| + | Restarting services... |
||
| − | Muu hulgas toetab see seade iwarp ja rocev2 protokolle, devlink vaatest paistab see nii |
||
| − | <pre> |
||
| − | # devlink dev param |
||
| − | pci/0000:81:00.0: |
||
| − | name enable_roce type generic |
||
| − | values: |
||
| − | cmode runtime value true |
||
| − | name enable_iwarp type generic |
||
| − | values: |
||
| − | cmode runtime value false |
||
| − | pci/0000:81:00.1: |
||
| − | name enable_roce type generic |
||
| − | values: |
||
| − | cmode runtime value false |
||
| − | name enable_iwarp type generic |
||
| − | values: |
||
| − | cmode runtime value true |
||
| − | </pre> |
||
| − | |||
| − | Korraga saab olla aktiivne üks või teine, nende vahel valmine toimub nt selliselt devlink abil |
||
| − | |||
| − | <pre> |
||
| − | # devlink dev param set pci/0000:81:00.1 name enable_iwarp value false cmode runtime |
||
| − | </pre> |
||
| − | |||
| − | rocev2 kasutamiseks on asjakohen süsteemidesse paigaldada omajagu rdma ja infiniband traditsiooniga tarkvara, nt |
||
| − | |||
| − | <pre> |
||
| − | apt-get install rdma-core |
||
| − | apt-get install perftest |
||
| − | apt-get install ibverbs-utils |
||
| − | apt-get install infiniband-diags |
||
| − | </pre> |
||
| − | |||
| − | Asjasse puutuvad sellised driverid |
||
| − | |||
| − | * ice - põhi draiver |
||
| − | * irdma - intel võrgukaartide (2024 aastal e810 ja midagi vanemat ka) - 'intel rdma' |
||
| − | |||
| − | Seadmed kui rdma seadmed paistavad |
||
| − | |||
| − | <pre> |
||
| − | root@pve-01:~# rdma link |
||
| − | link rocep129s0f0/1 state ACTIVE physical_state LINK_UP netdev enp129s0f0np0 |
||
| − | link rocep129s0f1/1 state ACTIVE physical_state LINK_UP netdev enp129s0f1np1 |
||
| − | </pre> |
||
| − | |||
| − | Kahe otse kaabliga süsteemi uurimiseks sobib öelda ühes ja teises arvutis nt nii, pve-02 on nö rping server ja pve-01 on rping klient; seejuures on iseloomulik, et tavalise võrguliidese peal tavalisel viisil võrku pealt kuulates ei ole midagi näha |
||
| − | |||
| − | <pre> |
||
| − | root@pve-02:~# rping -s -a 10.47.218.226 -v |
||
| − | server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr |
||
| − | server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs |
||
| − | server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst |
||
| − | server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu |
||
| − | server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv |
||
| − | server DISCONNECT EVENT... |
||
| − | wait for RDMA_READ_ADV state 10 |
||
| − | |||
| − | root@pve-01:~# rping -c -a 10.47.218.226 -v -C 5 |
||
| − | ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr |
||
| − | ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs |
||
| − | ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst |
||
| − | ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu |
||
| − | ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv |
||
| − | client DISCONNECT EVENT... |
||
| − | </pre> |
||
| − | |||
| − | Veel huvitavaid utiliite, seejuures on esimene seade 'devlink dev param ...' abil seadistatud rocev2 režiimi ja teine iwarp režiimi |
||
| − | |||
| − | <pre> |
||
| − | # ibv_devices |
||
| − | device node GUID |
||
| − | ------ ---------------- |
||
| − | rocep129s0f0 3eeceffffee667b2 |
||
| − | iwp129s0f1 3eeceffffee667b3 |
||
| − | |||
| − | |||
| − | # ibv_devinfo |
||
| − | hca_id: rocep129s0f0 |
||
| − | transport: InfiniBand (0) |
||
| − | fw_ver: 1.71 |
||
| − | node_guid: 3eec:efff:fee6:67b2 |
||
| − | sys_image_guid: 3eec:efff:fee6:67b2 |
||
| − | vendor_id: 0x8086 |
||
| − | vendor_part_id: 5531 |
||
| − | hw_ver: 0x2 |
||
| − | phys_port_cnt: 1 |
||
| − | port: 1 |
||
| − | state: PORT_ACTIVE (4) |
||
| − | max_mtu: 4096 (5) |
||
| − | active_mtu: 1024 (3) |
||
| − | sm_lid: 0 |
||
| − | port_lid: 1 |
||
| − | port_lmc: 0x00 |
||
| − | link_layer: Ethernet |
||
| − | |||
| − | hca_id: iwp129s0f1 |
||
| − | transport: iWARP (1) |
||
| − | fw_ver: 1.71 |
||
| − | node_guid: 3eec:efff:fee6:67b3 |
||
| − | sys_image_guid: 3eec:efff:fee6:67b3 |
||
| − | vendor_id: 0x8086 |
||
| − | vendor_part_id: 5531 |
||
| − | hw_ver: 0x2 |
||
| − | phys_port_cnt: 1 |
||
| − | port: 1 |
||
| − | state: PORT_ACTIVE (4) |
||
| − | max_mtu: 4096 (5) |
||
| − | active_mtu: 1024 (3) |
||
| − | sm_lid: 0 |
||
| − | port_lid: 1 |
||
| − | port_lmc: 0x00 |
||
| − | link_layer: Ethernet |
||
| − | </pre> |
||
| − | |||
| − | Töötaval juhul roce seadmega |
||
| − | |||
| − | <pre> |
||
| − | root@pve-02:~# ib_send_bw -i 1 -d rocep129s0f1 |
||
| − | |||
| − | ************************************ |
||
| − | * Waiting for client to connect... * |
||
| − | ************************************ |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | Send BW Test |
||
| − | Dual-port : OFF Device : rocep129s0f1 |
||
| − | Number of qps : 1 Transport type : IB |
||
| − | Connection type : RC Using SRQ : OFF |
||
| − | PCIe relax order: ON |
||
| − | ibv_wr* API : OFF |
||
| − | RX depth : 512 |
||
| − | CQ Moderation : 1 |
||
| − | Mtu : 1024[B] |
||
| − | Link type : Ethernet |
||
| − | GID index : 1 |
||
| − | Max inline data : 0[B] |
||
| − | rdma_cm QPs : OFF |
||
| − | Data ex. method : Ethernet |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | local address: LID 0x01 QPN 0x0004 PSN 0x807cd5 |
||
| − | GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:47:218:226 |
||
| − | remote address: LID 0x01 QPN 0x0004 PSN 0xba4fee |
||
| − | GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:47:218:225 |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] |
||
| − | Conflicting CPU frequency values detected: 3696.643000 != 3000.000000. CPU Frequency is not max. |
||
| − | 65536 1000 0.00 2762.05 0.044193 |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | </pre> |
||
| − | |||
| − | ja esimene arvuti kliendina |
||
| − | |||
| − | <pre> |
||
| − | root@pve-01:~# ib_send_bw -i 1 10.47.218.226 -d rocep129s0f1 |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | Send BW Test |
||
| − | Dual-port : OFF Device : rocep129s0f1 |
||
| − | Number of qps : 1 Transport type : IB |
||
| − | Connection type : RC Using SRQ : OFF |
||
| − | PCIe relax order: ON |
||
| − | ibv_wr* API : OFF |
||
| − | TX depth : 128 |
||
| − | CQ Moderation : 1 |
||
| − | Mtu : 1024[B] |
||
| − | Link type : Ethernet |
||
| − | GID index : 1 |
||
| − | Max inline data : 0[B] |
||
| − | rdma_cm QPs : OFF |
||
| − | Data ex. method : Ethernet |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | local address: LID 0x01 QPN 0x0004 PSN 0xba4fee |
||
| − | GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:47:218:225 |
||
| − | remote address: LID 0x01 QPN 0x0004 PSN 0x807cd5 |
||
| − | GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:47:218:226 |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] |
||
| − | Conflicting CPU frequency values detected: 3698.021000 != 3000.000000. CPU Frequency is not max. |
||
| − | 65536 1000 2757.88 2757.84 0.044126 |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | </pre> |
||
| − | |||
| − | kus |
||
| − | |||
| − | * tundub, et ib_send_bw on infiniband põlvnevusega utiliit |
||
| − | * Transport type on IB ehk infiniband, st nii nagu roce protokolli tööpõhimõte on |
||
| − | * Link type on ethernet |
||
| − | |||
| − | Mittetöötav juhum - kuna kasutatakse iwarp seadmel |
||
| − | |||
| − | <pre> |
||
| − | root@pve-02:~# ib_send_bw -i 1 -d iwp129s0f1 |
||
| − | |||
| − | ************************************ |
||
| − | * Waiting for client to connect... * |
||
| − | ************************************ |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | Send BW Test |
||
| − | Dual-port : OFF Device : iwp129s0f1 |
||
| − | Number of qps : 1 Transport type : IW |
||
| − | Connection type : RC Using SRQ : OFF |
||
| − | PCIe relax order: ON |
||
| − | ibv_wr* API : OFF |
||
| − | RX depth : 512 |
||
| − | CQ Moderation : 1 |
||
| − | Mtu : 1024[B] |
||
| − | Link type : Ethernet |
||
| − | GID index : 0 |
||
| − | Max inline data : 0[B] |
||
| − | rdma_cm QPs : OFF |
||
| − | Data ex. method : Ethernet |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | ethernet_read_keys: Couldn't read remote address |
||
| − | Unable to read to socket/rdma_cm |
||
| − | Failed to exchange data between server and clients |
||
| − | Failed to deallocate PD - Device or resource busy |
||
| − | Failed to destroy resources |
||
| − | </pre> |
||
| − | |||
| − | Mittetöötav juhtum - kuna seade on küll arvutis olemas, aga seal ei ole kinnitatud kõnealust ip aadressi |
||
| − | |||
| − | <pre> |
||
| − | root@pve-02:~# ib_send_bw -i 1 -d rocep129s0f0 |
||
| − | |||
| − | ************************************ |
||
| − | * Waiting for client to connect... * |
||
| − | ************************************ |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | Send BW Test |
||
| − | Dual-port : OFF Device : rocep129s0f0 |
||
| − | Number of qps : 1 Transport type : IB |
||
| − | Connection type : RC Using SRQ : OFF |
||
| − | PCIe relax order: ON |
||
| − | ibv_wr* API : OFF |
||
| − | RX depth : 512 |
||
| − | CQ Moderation : 1 |
||
| − | Mtu : 1024[B] |
||
| − | Link type : Ethernet |
||
| − | GID index : 0 |
||
| − | Max inline data : 0[B] |
||
| − | rdma_cm QPs : OFF |
||
| − | Data ex. method : Ethernet |
||
| − | --------------------------------------------------------------------------------------- |
||
| − | Failed to modify QP 5 to RTR |
||
| − | Unable to Connect the HCA's through the link |
||
| − | </pre> |
||
| − | |||
| − | Kõige praktilisem kasutusjuht on iscsi kasutamine iser + rocev2 viisil. Tulemusena on andmevahetuse kiirus 2x suurem võrreldes nö tavalisega. Target näeb välja selline |
||
| − | |||
| − | <pre> |
||
| − | /> ls / |
||
| − | o- / ......................................................................................................................... [...] |
||
| − | o- backstores .............................................................................................................. [...] |
||
| − | | o- block .................................................................................................. [Storage Objects: 1] |
||
| − | | | o- iscsi_block_md127 ............................................................. [/dev/md127 (27.9TiB) write-thru activated] |
||
| − | | | o- alua ................................................................................................... [ALUA Groups: 1] |
||
| − | | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] |
||
| − | | o- fileio ................................................................................................. [Storage Objects: 0] |
||
| − | | o- pscsi .................................................................................................. [Storage Objects: 0] |
||
| − | | o- ramdisk ................................................................................................ [Storage Objects: 0] |
||
| − | o- iscsi ............................................................................................................ [Targets: 1] |
||
| − | | o- iqn.2003-01.org.setup.lun.test .................................................................................... [TPGs: 1] |
||
| − | | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl] |
||
| − | | o- acls .......................................................................................................... [ACLs: 1] |
||
| − | | | o- iqn.1993-08.org.debian:01:b65e1ba35869 ................................................... [1-way auth, Mapped LUNs: 1] |
||
| − | | | o- mapped_lun0 ..................................................................... [lun0 block/iscsi_block_md127 (rw)] |
||
| − | | o- luns .......................................................................................................... [LUNs: 1] |
||
| − | | | o- lun0 ........................................................ [block/iscsi_block_md127 (/dev/md127) (default_tg_pt_gp)] |
||
| − | | o- portals .................................................................................................... [Portals: 1] |
||
| − | | o- 10.47.218.226:3261 ............................................................................................. [iser] |
||
| − | o- loopback ......................................................................................................... [Targets: 0] |
||
| − | o- srpt ............................................................................................................. [Targets: 0] |
||
| − | o- vhost ............................................................................................................ [Targets: 0] |
||
| − | o- xen-pvscsi ....................................................................................................... [Targets: 0] |
||
| − | </pre> |
||
| − | |||
| − | ning initiator |
||
| − | |||
| − | <pre> |
||
| − | root@pve-01:~# iscsiadm -m discovery -t st -p 10.47.218.226:3261 |
||
| − | root@pve-01:~# iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -p 10.47.218.226:3261 -o update -n iface.transport_name -v iser |
||
| − | root@pve-01:~# iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -p 10.47.218.226:3261 -l |
||
| − | root@pve-01:~# lsscsi -s |
||
... |
... |
||
| − | |||
| − | root@pve-01:~# iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -p 10.47.218.226:3261 -u |
||
| − | root@pve-01:~# iscsiadm -m discoverydb -t sendtargets -p 10.47.218.226:3261 -o delete |
||
| − | root@pve-01:~# iscsiadm -m discoverydb |
||
</pre> |
</pre> |
||
| − | |||
| − | Kasulikud lisamaterjalid |
||
| − | |||
| − | * 'Intel E800 devices now support iWARP and RoCE protocols' - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/8.7_release_notes/new-features |
||
| − | * https://forum.proxmox.com/threads/mellanox-connectx-4-lx-and-brigde-vlan-aware-on-proxmox-8-0-1.130902/ |
||
| − | * https://docs.nvidia.com/networking/display/mftv4270/updating+firmware+using+ethtool/devlink+and+-mfa2+file |
||
| − | * https://edc.intel.com/content/www/ca/fr/design/products/ethernet/appnote-e810-eswitch-switchdev-mode-config-guide/eswitch-mode-switchdev-and-legacy/ |
||
| − | * nvidia-mlnx-ofed-documentation-v23-07.pdf |
||
| − | * https://enterprise-support.nvidia.com/s/article/How-To-Enable-Verify-and-Troubleshoot-RDMA |
||
| − | * https://enterprise-support.nvidia.com/s/article/howto-configure-lio-enabled-with-iser-for-rhel7-inbox-driver |
||
| − | * https://dri.freedesktop.org/docs/drm/networking/devlink/ice.html |
||
| − | |||
| − | ===2024 kevad täiendused - VF=== |
||
| − | |||
| − | TODO |
||
| − | |||
| − | Kasulikud lisamaterjalid |
||
| − | |||
| − | * https://clouddocs.f5.com/cloud/public/v1/kvm/kvm_intel.html |
||
| − | * https://edc.intel.com/content/www/ca/fr/design/products/ethernet/appnote-e810-eswitch-switchdev-mode-config-guide/configuration-steps/ |
||
| − | * https://markmcb.com/freebsd/vs_linux/sriov_is_first_class/ |
||
| − | * https://serverfault.com/questions/931228/error-in-enbling-sr-iov-with-ubuntu-18-04-on-intel-ixgbe-intel-x550-dell-r64 |
||
| − | * https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/pdf/configuring_and_managing_virtualization/red_hat_enterprise_linux-9-configuring_and_managing_virtualization-en-us.pdf |
||
| − | * https://docs.nvidia.com/networking/display/mlnxofedv23100550/kubernetes+using+sr-iov |
||
| − | * https://www.supermicro.com/support/faqs/faq.cfm?faq=31936 |
||
| − | * https://www.sealingtech.com/blog/sr-iov-and-promiscuous-mode/ |
||
| − | * https://powercampus.de/en/adding-logical-sr-iov-ports/ |
||
| − | * https://enterprise-support.nvidia.com/s/article/howto-configure-lag-in-the-vm-over-sr-iov |
||
| − | |||
| − | ===2024 kevad täiendused - Mellanox/NVidia=== |
||
| − | |||
| − | TODO |
||
| − | |||
| − | * https://network.nvidia.com/pdf/prod_software/Ubuntu_20_04_Inbox_Driver_User_Manual.pdf |
||
===Kasulikud lisamaterjalid=== |
===Kasulikud lisamaterjalid=== |
||
Viimane redaktsioon: 16. detsember 2024, kell 02:18
Sissejuhatus
TODO
Tööpõhimõte
TODO
Riistvara
Et toeta üldse, st nö pole lootust ka
TODO
Võrgukaardi mudel põhimõttelist toetab, mis ei tähenda, et praktiliselt kasutada saab (nt funktsionaalsust sisselülitav riistvaraline võti on puudu, punni kujuline)
# lspci | grep -i ether 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12) 07:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12) 16:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12) 18:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)
Toetab põhimõtteliselt, aga midagi siiski puudub
[17763.496402] bnx2i: LOM is not enable to offload iSCSI connections, dev=eth0
Redhat v. 7
iSCSI initiatori tarkvara paigaldamiseks sobib öelda
# yum install iscsi-initiator-utils # yum install iscsi-initiator-utils-iscsiuio
Tulemusena tekivad failisüsteemi
- /etc/iscsi/iscsid.conf - peamine seadistusfail
- /usr/sbin/iscsid - Open-iSCSI daemon
- /usr/sbin/iscsiadm - open-iscsi administration utility
- /usr/sbin/iscsiuio - iSCSI UserSpace I/O driver
Offload jt liideste nimekirja esitamine
# iscsiadm -m iface default tcp,<empty>,<empty>,<empty>,<empty> iser iser,<empty>,<empty>,<empty>,<empty> bnx2i.78:e7:d1:f6:07:77 bnx2i,78:e7:d1:f6:07:77,<empty>,<empty>,<empty> bnx2i.78:e7:d1:f6:07:75 bnx2i,78:e7:d1:f6:07:75,<empty>,<empty>,<empty>
Offload liidesele ip aadressi omistamine (tavalisel ethernet liidesel on reeglina mac ühe võrra väiksema viimase numbriga ning ip aadress seadistatud sarnaselt ühe võrra väiksem)
# iscsiadm -m iface -I bnx2i.78:e7:d1:f6:07:75 -o update -n iface.ipaddress -v 10.0.0.211
Discovery üle offload liidese
# iscsiadm -m discovery -t st -p 10.0.0.209:3260 -I bnx2i.78:e7:d1:f6:07:75 -P 1
Target: iqn.201410.imool.ee.test1:test1
Portal: 10.0.0.209:3260,1
Iface Name: bnx2i.78:e7:d1:f6:07:75
Külgeühendumiseks sobib öelda
# iscsiadm -m node -T iqn.201410.imool.ee.test1:test1 -p 10.0.0.209:3260 -l Logging in to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.0.209,3260] (multiple) Login to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.0.209,3260] successful.
Töötavale sesioonile vastab selline väljund
# iscsiadm -m session -P 1
Target: iqn.201410.imool.ee.test1:test1 (non-flash)
Current Portal: 10.0.0.209:48140,1
Persistent Portal: 10.0.0.209:3260,1
**********
Interface:
**********
Iface Name: bnx2i.78:e7:d1:f6:07:75
Iface Transport: bnx2i
Iface Initiatorname: iqn.1994-05.com.redhat:7fe6981fef65
Iface IPaddress: 10.0.0.211
Iface HWaddress: 78:e7:d1:f6:07:75
Iface Netdev: enp2s0f0
SID: 3
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
Kui selliselt tekkinud plokkseadet kasutada ning võrku kõrvalt kuulata targetis, siis on näha, et pöördutakse .211 st offload komponendile omistatud ip aadressilt.
Töötavad protsessid
# ps aux | grep iscsi .. root 2736 0.0 0.0 13832 512 ? Ss 01:41 0:00 /usr/sbin/iscsid root 2737 0.0 0.1 16480 4220 ? S<Ls 01:41 0:00 /usr/sbin/iscsid root 2740 5.7 1.0 248492 43640 ? S<Lsl 01:41 2:15 /usr/sbin/iscsiuio ...
dmesg väljundis paistab taustal midagi sellist
[ 1071.887882] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.19 (December 19, 2013) [ 1071.891303] bnx2 0000:02:00.0 enp2s0f0: Added CNIC device [ 1071.895742] bnx2 0000:02:00.1 enp2s0f1: Added CNIC device [ 1071.905159] Loading iSCSI transport class v2.0-870. [ 1071.913716] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.6.2 (Jun 06, 2013) [ 1071.917250] iscsi: registered transport (bnx2i) [ 1071.920896] scsi3 : Broadcom Offload iSCSI Initiator [ 1071.924833] bnx2i [02:00.01]: ISCSI_INIT passed [ 1071.928984] scsi4 : Broadcom Offload iSCSI Initiator [ 1071.932802] bnx2i [02:00.00]: ISCSI_INIT passed [ 1249.378334] connection1:0: bnx2i: conn update - MBL 0xfffc00 FBL 0x40000MRDSL_I 0x2000 MRDSL_T 0x2000 [ 1423.291754] connection2:0: bnx2i: conn update - MBL 0x40000 FBL 0x10000MRDSL_I 0x40000 MRDSL_T 0x2000
Asjassepuutuvad failid
# find /var/lib/iscsi -type f -ls 9496443 4 -rw------- 1 root root 474 Feb 10 01:38 /var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:77 9496444 4 -rw------- 1 root root 506 Feb 10 01:40 /var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:75 4196315 4 -rw------- 1 root root 2127 Feb 10 02:18 /var/lib/iscsi/nodes/iqn.201410.piia.ee.test1:test1/10.0.0.209,3260,1/bnx2i.78:e7:d1:f6:07:75 1088931 4 -rw------- 1 root root 552 Feb 10 02:18 /var/lib/iscsi/send_targets/10.0.0.209,3260/st_config
kus
# cat '/var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:75' # BEGIN RECORD 6.2.0.873-21 iface.iscsi_ifacename = bnx2i.78:e7:d1:f6:07:75 iface.ipaddress = 10.0.0.211 iface.hwaddress = 78:e7:d1:f6:07:75 iface.transport_name = bnx2i iface.vlan_id = 0 iface.vlan_priority = 0 iface.iface_num = 0 iface.mtu = 0 iface.port = 0 iface.tos = 0 iface.ttl = 0 iface.tcp_wsf = 0 iface.tcp_timer_scale = 0 iface.def_task_mgmt_timeout = 0 iface.erl = 0 iface.max_receive_data_len = 0 iface.first_burst_len = 0 iface.max_outstanding_r2t = 0 iface.max_burst_len = 0 # END RECORD
Kasulikud lisamaterjalid
SupportProduct -> Documentation -> Red Hat Enterprise Linux 7 -> Storage Administration Guide -> 26.12. CONFIGURING ISCSI OFFLOAD AND INTERFACE BINDING
Ubuntu v. 14.04
TODO
Debian v. 8 Jessie
Riistvara, BIOS
Broadcom NetXtreme II Ethernet Boot Agent v5.0.5 <F9 = Setup> Copyright (C) 2000-2009 Broadcom Corporation All rights reserved. Press Ctrl-S to Configure Device (MAC Address - 78E7D1F60774)
Seadmed
# lspci | grep -i ether 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
bnx2i draiveri laadimisel öeldakse dmesgi
[ 117.382139] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.20 (March 14, 2014) [ 117.382275] bnx2 0000:02:00.0 eth0: Added CNIC device [ 117.383534] bnx2 0000:02:00.1 eth1: Added CNIC device [ 117.397584] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.6.2 (Jun 06, 2013) [ 117.397726] iscsi: registered transport (bnx2i) [ 117.401852] scsi3 : Broadcom Offload iSCSI Initiator [ 117.405854] scsi4 : Broadcom Offload iSCSI Initiator
# iscsiadm -m iface default tcp,<empty>,<empty>,<empty>,<empty> iser iser,<empty>,<empty>,<empty>,<empty> bnx2i.00:00:00:00:00:00 bnx2i,00:00:00:00:00:00,<empty>,<empty>,<empty> bnx2i.10:e7:d1:f6:07:75 bnx2i,10:e7:d1:f6:07:75,10.0.1.211,<empty>,<empty>
ip aadressi omistamine
# iscsiadm -m iface -I bnx2i.00:e7:d1:f6:07:75 -o update -n iface.ipaddress -v 10.0.1.209
tulemusena on
# cat /etc/iscsi/ifaces/bnx2i.10\:e7\:d1\:f6\:07\:75 # BEGIN RECORD 2.0-873 iface.iscsi_ifacename = bnx2i.10:e7:d1:f6:07:75 iface.ipaddress = 10.0.1.211 iface.hwaddress = 10:e7:d1:f6:07:75 iface.transport_name = bnx2i iface.vlan_id = 0 iface.vlan_priority = 0 iface.iface_num = 0 iface.mtu = 0 iface.port = 0
Discovery tegemiseks
# /usr/bin/iscsiadm -I bnx2i.00:e7:d1:f6:07:75 -m discovery -t sendtargets -p 10.0.1.209 iscsiadm: can not connect to iSCSI daemon (111)! iscsiadm: uIP daemon is not up iscsiadm: Could not set host net params (err 20)
Debian v.6 Squeeze
Discovery bnx2i kaudu
# /usr/bin/iscsiadm -I bnx2i.78:e7:d1:f6:07:75 -m discovery -t sendtargets -p 10.0.1.209 10.0.1.209:3260,1 iqn.201410.imool.ee.test1:test1
Login
# iscsiadm -m node -I bnx2i.78:e7:d1:f6:07:75 -T iqn.201410.imool.ee.test1:test1 -p 10.0.1.209:3260 -l Logging in to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.1.209,3260] iscsiadm: Could not login to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.1.209,3260]: iscsiadm: initiator reported error (4 - encountered connection failure)
FCoE
Kasulikud lisamaterjalid
- http://en.community.dell.com/techcenter/b/techcenter/archive/2012/07/24/red-hat-enterprise-linux-6-3-on-dell-poweredge-servers-features-and-native-drivers
- http://www.qlogic.com/OEMPartnerships/Dell/Documents/Converged/DataSheet_57810SDell_CNA_DualPort_10GBASE-T.pdf
- http://forums.qlogic.com/SupportCenter/Customer_support_main?id=kA28000000008bwCAA
- http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c04448818-3.pdf
LIO target
TODO
Tundub, et 2018 sügisel on LIO (Linux-IO) kõige õigem lahendus iSCSI protokolli abil plokkseadme ressurssi üle nö tavalise ethernet võrgu kasutamiseks. Iseenesest LIO võimaldab teha muudki, LIO sisaldab
- erinevate back storage toetust (fail, partitsioon, lvm jt )
- erinevate andmevahetuse kanalite toetust (ethernet, fc jt)
Tööpõhimõte
LIO töötamisega ja sealhulgas võrgus nt iscsi teenuse pakkumisega ei kaasneb ühtegi user-space protsessi; tööd teeb kernel ja kerneli moodulid. iSCSI lahenduse määratlevad sellised objektid ja nende vahelised seosed
- plokkseade
- target portal
- lun
- acl
- initiatior (klient)
LIO target seadistatakse sysfs failisüsteemis /sys/kernel/config/target alla jääva osaga töötades. Muudatuste haldamist lihtsustab targetcli utiliit.
Tarkvara paigaldamine
2018 sügisel sobib Debian v. 9 keskkonnas öelda
# apt-get install targetcli-fb
CentOS v. 7 keskkonnas sobib öelda
# yum install targetcli
Seadistamine
Ettevalmistused
- vajalik on teada iscsi initiator arvuti iqn väärtust (olgu see nt iqn.1993-08.org.debian:01:aaaaaaaaaaa)
# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1993-08.org.debian:01:aaaaaaaaaaa
- antud näites ei piirata ligipääsu iscsi ressursile parooliga, ainult kliendi iqn väärtusega (täindava ligipääsu saab seadistada nt paketifiltriga)
Käsuga
# targetcli
jõuab sysfs failisüsteemi haldamisse interaktiivsesse režiimi.
- plokkseadme kirjeldamine
/backstores/block create iscsi_block_1 /dev/vdb
- target iqn kirjeldamine ja portal kirjeldamine
/iscsi create iqn.2018-12.moraal.srv:storage.target03
- iqn kirjeldamine, millelt on lubatud ligi pääseda
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls create iqn.1993-08.org.debian:01:aaaaaaaaaaa
- lun tekitamine ja sidumine targetiga
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/luns create /backstores/block/iscsi_block_1
Paistab, et peale ülaltoodud käskude ütlemist muudatused kehtestatakse. Seadistused salvesatakse faili
/etc/rtslib-fb-target/saveconfig.json
Misc
- iqn eemaldamine, millelt on lubatud ligi pääseda
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls delete iqn.1993-08.org.debian:01:aaaaaaaaaaa
- portal eemaldamine (praktiliselt lõpetatakse ka 0.0.0.0:3260/tcp kuulamine)
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls delete 0.0.0.0 3260
- target ja block seade eemaldamine
/iscsi> /iscsi delete iqn.2021-01.tartu:storage.target01 /backstores> /backstores/fileio delete disc2_sda
Haldamine
root@cifs:~# targetcli sessions alias: pm-moraal-4-v52 sid: 1 type: Normal session-state: LOGGED_IN
LIO teenust käivitab
# systemctl status rtslib-fb-targetctl
LIO'ga tegelevad kerneli moodulid
# lsmod | grep ^target target_core_user 20480 0 target_core_pscsi 20480 0 target_core_file 20480 2 target_core_iblock 20480 2 ...
Võrgus teenindab kernel-space
# netstat -lnpt | grep -- - Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN -
Tundub, et Proxmox peal saab kergesti hakata iscsi ressursse kaustama, aga kaustamist ära lõpetada polegi webgui kaudu võimalik, käsurealt sobib öelda
# iscsiadm -m session -P 1 # iscsiadm -m node -T iqn.2018-12.moraal.srv:storage.target02 -p 192.168.10.127 -u # iscsiadm -m node -T iqn.2018-12.moraal.srv:storage.target02 -p 192.168.10.127:3260 -o delete
Proxmox pealt kasutamiseks võib olla tark esmalt host peal /etc/lvm/lvm.conf peale seadistada filter ja global_filter
TODO
Kasulikud lisamaterjalid
- http://www.linux-iscsi.org/Doc/LIO%20Admin%20Manual.pdf
- https://wiki.rvijay.in/index.php/Configuring_iSCSI_target_using_'targetcli'
- http://linux-iscsi.org/wiki/ISCSI
- https://www.server-world.info/en/note?os=Debian_9&p=iscsi&f=1
- https://www.thegeekdiary.com/how-to-configure-iscsi-target-and-initiator-using-targetcli-in-centos-rhel-7/
- https://www.linuxhelp.com/how-to-create-and-setup-luns-using-lvm
2024 kevad täiendused - iSER (iSCSI Extensions for RDMA)
Vt https://www.auul.pri.ee/wiki/Intel_E810
Ubuntu 24.04 ja targetcli-fb
root@tgt-zfs-over-iscsi:~# apt-get install targetcli-fb
..
Unpacking targetcli-fb (1:2.1.53-1ubuntu3) ...
Setting up python3-wcwidth (0.2.5+dfsg1-1.1ubuntu1) ...
Setting up python3-typing-extensions (4.10.0-1) ...
Setting up python3-pyudev (0.24.0-1) ...
Setting up python3-rtslib-fb (2.1.74-0ubuntu5) ...
Created symlink /etc/systemd/system/multi-user.target.wants/rtslib-fb-targetctl.service → /usr/lib/systemd/system/rtslib-fb-targetctl.service.
Setting up python3-urwid (2.6.10-1) ...
Setting up python3-configshell-fb (1:1.1.28-2.1) ...
/usr/lib/python3/dist-packages/configshell/console.py:39: SyntaxWarning: invalid escape sequence '\['
_re_ansi_seq = re.compile('(\033\[..?m)')
/usr/lib/python3/dist-packages/configshell/shell.py:116: SyntaxWarning: invalid escape sequence '\+'
var = Word(alphanums + '?;&*$!#,=_\+/.<>()~@:-%[]')
/usr/lib/python3/dist-packages/configshell/shell.py:118: SyntaxWarning: invalid escape sequence '\-'
keyword = Word(alphanums + '_\-')
/usr/lib/python3/dist-packages/configshell/shell.py:124: SyntaxWarning: invalid escape sequence '\['
pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \
/usr/lib/python3/dist-packages/configshell/shell.py:124: SyntaxWarning: invalid escape sequence '\['
pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \
/usr/lib/python3/dist-packages/configshell/shell.py:131: SyntaxWarning: invalid escape sequence '\|'
readline.set_completer_delims('\t\n ~!#$^&(){}\|;\'",?')
/usr/lib/python3/dist-packages/configshell_fb/console.py:39: SyntaxWarning: invalid escape sequence '\['
_re_ansi_seq = re.compile('(\033\[..?m)')
/usr/lib/python3/dist-packages/configshell_fb/shell.py:116: SyntaxWarning: invalid escape sequence '\+'
var = Word(alphanums + '?;&*$!#,=_\+/.<>()~@:-%[]')
/usr/lib/python3/dist-packages/configshell_fb/shell.py:118: SyntaxWarning: invalid escape sequence '\-'
keyword = Word(alphanums + '_\-')
/usr/lib/python3/dist-packages/configshell_fb/shell.py:124: SyntaxWarning: invalid escape sequence '\['
pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \
/usr/lib/python3/dist-packages/configshell_fb/shell.py:124: SyntaxWarning: invalid escape sequence '\['
pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \
/usr/lib/python3/dist-packages/configshell_fb/shell.py:131: SyntaxWarning: invalid escape sequence '\|'
readline.set_completer_delims('\t\n ~!#$^&(){}\|;\'",?')
Setting up targetcli-fb (1:2.1.53-1ubuntu3) ...
/usr/lib/python3/dist-packages/targetcli/ui_root.py:161: SyntaxWarning: invalid escape sequence '\s'
backups = [line for line in prefs.read().splitlines() if re.match('^max_backup_files\s*=', line)]
Processing triggers for man-db (2.12.0-4build2) ...
Scanning processes...
Scanning candidates...
Scanning linux images...
Running kernel seems to be up-to-date.
Restarting services...
...
Kasulikud lisamaterjalid
- http://h10032.www1.hp.com/ctg/Manual/c01555907.pdf
- https://github.com/mikechristie/open-iscsi
- http://www.dell.com/downloads/global/power/ps4q08-20080208-Broadcom.pdf
- http://www.broadcom.com/collateral/pb/5709C-PB02-R.pdf
- http://www.broadcom.com/collateral/pb/5719-PB01-R.pdf
- http://www.broadcom.com/collateral/pb/5720_Chip_PB.pdf
- http://forum.broadcom.com/showthread.php?91157-Broadcom-NetXtreme-II-1Gb-and-10Gb-Products-Now-Serviced-by-QLogic
- http://www.broadcom.com/support/ethernet_nic/downloaddrivers.php
- http://www.qlogic.com/solutions/Pages/Ethernet-NIC-FAQs.aspx
