ISCSI offload

Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti

Sissejuhatus

TODO

Tööpõhimõte

TODO

Riistvara

Et toeta üldse, st nö pole lootust ka 

TODO

Võrgukaardi mudel põhimõttelist toetab, mis ei tähenda, et praktiliselt kasutada saab (nt funktsionaalsust sisselülitav riistvaraline võti on puudu, punni kujuline) 

# lspci | grep -i ether
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)
07:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)
16:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)
18:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)

Toetab põhimõtteliselt, aga midagi siiski puudub 

[17763.496402] bnx2i: LOM is not enable to offload iSCSI connections, dev=eth0

Redhat v. 7

iSCSI initiatori tarkvara paigaldamiseks sobib öelda 

# yum install iscsi-initiator-utils
# yum install iscsi-initiator-utils-iscsiuio

Tulemusena tekivad failisüsteemi

  • /etc/iscsi/iscsid.conf - peamine seadistusfail
  • /usr/sbin/iscsid - Open-iSCSI daemon
  • /usr/sbin/iscsiadm - open-iscsi administration utility
  • /usr/sbin/iscsiuio - iSCSI UserSpace I/O driver

Offload jt liideste nimekirja esitamine 

# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
bnx2i.78:e7:d1:f6:07:77 bnx2i,78:e7:d1:f6:07:77,<empty>,<empty>,<empty>
bnx2i.78:e7:d1:f6:07:75 bnx2i,78:e7:d1:f6:07:75,<empty>,<empty>,<empty>

Offload liidesele ip aadressi omistamine (tavalisel ethernet liidesel on reeglina mac ühe võrra väiksema viimase numbriga ning ip aadress seadistatud sarnaselt ühe võrra väiksem) 

# iscsiadm -m iface -I bnx2i.78:e7:d1:f6:07:75 -o update -n iface.ipaddress -v 10.0.0.211

Discovery üle offload liidese 

# iscsiadm -m discovery -t st -p 10.0.0.209:3260 -I bnx2i.78:e7:d1:f6:07:75 -P 1
Target: iqn.201410.imool.ee.test1:test1
       Portal: 10.0.0.209:3260,1
               Iface Name: bnx2i.78:e7:d1:f6:07:75

Külgeühendumiseks sobib öelda 

# iscsiadm -m node -T iqn.201410.imool.ee.test1:test1 -p 10.0.0.209:3260 -l
Logging in to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.0.209,3260] (multiple)
Login to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.0.209,3260] successful.

Töötavale sesioonile vastab selline väljund 

# iscsiadm -m session -P 1
Target: iqn.201410.imool.ee.test1:test1 (non-flash)
       Current Portal: 10.0.0.209:48140,1
       Persistent Portal: 10.0.0.209:3260,1
               **********
               Interface:
               **********
               Iface Name: bnx2i.78:e7:d1:f6:07:75
               Iface Transport: bnx2i
               Iface Initiatorname: iqn.1994-05.com.redhat:7fe6981fef65
               Iface IPaddress: 10.0.0.211
               Iface HWaddress: 78:e7:d1:f6:07:75
               Iface Netdev: enp2s0f0
               SID: 3
               iSCSI Connection State: LOGGED IN
               iSCSI Session State: LOGGED_IN
               Internal iscsid Session State: NO CHANGE

Kui selliselt tekkinud plokkseadet kasutada ning võrku kõrvalt kuulata targetis, siis on näha, et pöördutakse .211 st offload komponendile omistatud ip aadressilt.

Töötavad protsessid 

# ps aux | grep iscsi
..
root      2736  0.0  0.0  13832   512 ?        Ss   01:41   0:00 /usr/sbin/iscsid
root      2737  0.0  0.1  16480  4220 ?        S<Ls 01:41   0:00 /usr/sbin/iscsid
root      2740  5.7  1.0 248492 43640 ?        S<Lsl 01:41   2:15 /usr/sbin/iscsiuio
...

dmesg väljundis paistab taustal midagi sellist 

[ 1071.887882] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.19 (December 19, 2013)
[ 1071.891303] bnx2 0000:02:00.0 enp2s0f0: Added CNIC device
[ 1071.895742] bnx2 0000:02:00.1 enp2s0f1: Added CNIC device
[ 1071.905159] Loading iSCSI transport class v2.0-870.
[ 1071.913716] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.6.2 (Jun 06, 2013)
[ 1071.917250] iscsi: registered transport (bnx2i)
[ 1071.920896] scsi3 : Broadcom Offload iSCSI Initiator
[ 1071.924833] bnx2i [02:00.01]: ISCSI_INIT passed
[ 1071.928984] scsi4 : Broadcom Offload iSCSI Initiator
[ 1071.932802] bnx2i [02:00.00]: ISCSI_INIT passed
[ 1249.378334]  connection1:0: bnx2i: conn update - MBL 0xfffc00 FBL 0x40000MRDSL_I 0x2000 MRDSL_T 0x2000 
[ 1423.291754]  connection2:0: bnx2i: conn update - MBL 0x40000 FBL 0x10000MRDSL_I 0x40000 MRDSL_T 0x2000

Asjassepuutuvad failid 

# find /var/lib/iscsi -type f -ls
9496443    4 -rw-------   1 root  root   474 Feb 10 01:38 /var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:77
9496444    4 -rw-------   1 root  root   506 Feb 10 01:40 /var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:75
4196315    4 -rw-------   1 root  root  2127 Feb 10 02:18 /var/lib/iscsi/nodes/iqn.201410.piia.ee.test1:test1/10.0.0.209,3260,1/bnx2i.78:e7:d1:f6:07:75
1088931    4 -rw-------   1 root  root   552 Feb 10 02:18 /var/lib/iscsi/send_targets/10.0.0.209,3260/st_config

kus 

# cat '/var/lib/iscsi/ifaces/bnx2i.78:e7:d1:f6:07:75'
# BEGIN RECORD 6.2.0.873-21
iface.iscsi_ifacename = bnx2i.78:e7:d1:f6:07:75
iface.ipaddress = 10.0.0.211
iface.hwaddress = 78:e7:d1:f6:07:75
iface.transport_name = bnx2i
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
iface.tos = 0
iface.ttl = 0
iface.tcp_wsf = 0
iface.tcp_timer_scale = 0
iface.def_task_mgmt_timeout = 0
iface.erl = 0
iface.max_receive_data_len = 0
iface.first_burst_len = 0
iface.max_outstanding_r2t = 0
iface.max_burst_len = 0
# END RECORD

Kasulikud lisamaterjalid 

SupportProduct -> Documentation -> Red Hat Enterprise Linux  7 -> Storage Administration Guide
  -> 26.12. CONFIGURING ISCSI OFFLOAD AND INTERFACE BINDING

Ubuntu v. 14.04

TODO

Debian v. 8 Jessie

Riistvara, BIOS 

Broadcom NetXtreme II Ethernet Boot Agent v5.0.5                  <F9 = Setup>
Copyright (C) 2000-2009 Broadcom Corporation
All rights reserved.
Press Ctrl-S to Configure Device (MAC Address - 78E7D1F60774)

Seadmed 

# lspci | grep -i ether
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

bnx2i draiveri laadimisel öeldakse dmesgi 

[  117.382139] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.20 (March 14, 2014)
[  117.382275] bnx2 0000:02:00.0 eth0: Added CNIC device
[  117.383534] bnx2 0000:02:00.1 eth1: Added CNIC device
[  117.397584] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.6.2 (Jun 06, 2013)
[  117.397726] iscsi: registered transport (bnx2i)
[  117.401852] scsi3 : Broadcom Offload iSCSI Initiator
[  117.405854] scsi4 : Broadcom Offload iSCSI Initiator

# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
bnx2i.00:00:00:00:00:00 bnx2i,00:00:00:00:00:00,<empty>,<empty>,<empty>
bnx2i.10:e7:d1:f6:07:75 bnx2i,10:e7:d1:f6:07:75,10.0.1.211,<empty>,<empty>

ip aadressi omistamine 

# iscsiadm -m iface -I bnx2i.00:e7:d1:f6:07:75 -o update -n iface.ipaddress -v 10.0.1.209

tulemusena on 

# cat /etc/iscsi/ifaces/bnx2i.10\:e7\:d1\:f6\:07\:75 
# BEGIN RECORD 2.0-873
iface.iscsi_ifacename = bnx2i.10:e7:d1:f6:07:75
iface.ipaddress = 10.0.1.211
iface.hwaddress = 10:e7:d1:f6:07:75
iface.transport_name = bnx2i
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0

Discovery tegemiseks 

# /usr/bin/iscsiadm -I bnx2i.00:e7:d1:f6:07:75 -m discovery -t sendtargets -p 10.0.1.209
iscsiadm: can not connect to iSCSI daemon (111)!
iscsiadm: uIP daemon is not up
iscsiadm: Could not set host net params (err 20)

Debian v.6 Squeeze

Discovery bnx2i kaudu 

# /usr/bin/iscsiadm -I bnx2i.78:e7:d1:f6:07:75 -m discovery -t sendtargets -p 10.0.1.209
10.0.1.209:3260,1 iqn.201410.imool.ee.test1:test1

Login 

# iscsiadm -m node -I bnx2i.78:e7:d1:f6:07:75 -T iqn.201410.imool.ee.test1:test1 -p 10.0.1.209:3260 -l
Logging in to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal: 10.0.1.209,3260]
iscsiadm: Could not login to [iface: bnx2i.78:e7:d1:f6:07:75, target: iqn.201410.imool.ee.test1:test1, portal:  10.0.1.209,3260]: 
iscsiadm: initiator reported error (4 - encountered connection failure)

FCoE

Kasulikud lisamaterjalid

LIO target

TODO

Tundub, et 2018 sügisel on LIO (Linux-IO) kõige õigem lahendus iSCSI protokolli abil plokkseadme ressurssi üle nö tavalise ethernet võrgu kasutamiseks. Iseenesest LIO võimaldab teha muudki, LIO sisaldab

  • erinevate back storage toetust (fail, partitsioon, lvm jt )
  • erinevate andmevahetuse kanalite toetust (ethernet, fc jt)

Tööpõhimõte

LIO töötamisega ja sealhulgas võrgus nt iscsi teenuse pakkumisega ei kaasneb ühtegi user-space protsessi; tööd teeb kernel ja kerneli moodulid. iSCSI lahenduse määratlevad sellised objektid ja nende vahelised seosed

  • plokkseade
  • target portal
  • lun
  • acl
  • initiatior (klient)

LIO target seadistatakse sysfs failisüsteemis /sys/kernel/config/target alla jääva osaga töötades. Muudatuste haldamist lihtsustab targetcli utiliit.

Tarkvara paigaldamine

2018 sügisel sobib Debian v. 9 keskkonnas öelda 

# apt-get install targetcli-fb

CentOS v. 7 keskkonnas sobib öelda 

# yum install targetcli

Seadistamine

Ettevalmistused

  • vajalik on teada iscsi initiator arvuti iqn väärtust (olgu see nt iqn.1993-08.org.debian:01:aaaaaaaaaaa)
# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.1993-08.org.debian:01:aaaaaaaaaaa
  • antud näites ei piirata ligipääsu iscsi ressursile parooliga, ainult kliendi iqn väärtusega (täindava ligipääsu saab seadistada nt paketifiltriga)

Käsuga 

# targetcli

jõuab sysfs failisüsteemi haldamisse interaktiivsesse režiimi.

  • plokkseadme kirjeldamine
/backstores/block create iscsi_block_1 /dev/vdb
  • target iqn kirjeldamine ja portal kirjeldamine
/iscsi create iqn.2018-12.moraal.srv:storage.target03
  • iqn kirjeldamine, millelt on lubatud ligi pääseda
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls create iqn.1993-08.org.debian:01:aaaaaaaaaaa
  • lun tekitamine ja sidumine targetiga
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/luns create /backstores/block/iscsi_block_1

Paistab, et peale ülaltoodud käskude ütlemist muudatused kehtestatakse. Seadistused salvesatakse faili 

/etc/rtslib-fb-target/saveconfig.json

Misc

  • iqn eemaldamine, millelt on lubatud ligi pääseda
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls delete iqn.1993-08.org.debian:01:aaaaaaaaaaa
  • portal eemaldamine (praktiliselt lõpetatakse ka 0.0.0.0:3260/tcp kuulamine)
/iscsi/iqn.2018-12.moraal.srv:storage.target03/tpg1/acls delete 0.0.0.0 3260
  • target ja block seade eemaldamine
/iscsi> /iscsi delete iqn.2021-01.tartu:storage.target01
/backstores> /backstores/fileio delete disc2_sda

Haldamine

root@cifs:~# targetcli sessions
alias: pm-moraal-4-v52	sid: 1 type: Normal session-state: LOGGED_IN

LIO teenust käivitab 

# systemctl status rtslib-fb-targetctl

LIO'ga tegelevad kerneli moodulid 

# lsmod | grep ^target
target_core_user       20480  0
target_core_pscsi      20480  0
target_core_file       20480  2
target_core_iblock     20480  2
...

Võrgus teenindab kernel-space 

# netstat -lnpt | grep -- -
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:3260            0.0.0.0:*               LISTEN      -

Tundub, et Proxmox peal saab kergesti hakata iscsi ressursse kaustama, aga kaustamist ära lõpetada polegi webgui kaudu võimalik, käsurealt sobib öelda 

# iscsiadm -m session -P 1
# iscsiadm -m node -T iqn.2018-12.moraal.srv:storage.target02 -p 192.168.10.127 -u
# iscsiadm -m node -T iqn.2018-12.moraal.srv:storage.target02  -p 192.168.10.127:3260 -o delete

Proxmox pealt kasutamiseks võib olla tark esmalt host peal /etc/lvm/lvm.conf peale seadistada filter ja global_filter 

TODO

Kasulikud lisamaterjalid

2024 kevad täiendused - iSER (iSCSI Extensions for RDMA)

Vt https://www.auul.pri.ee/wiki/Intel_E810

Ubuntu 24.04 ja targetcli-fb

root@tgt-zfs-over-iscsi:~# apt-get install targetcli-fb

..
Unpacking targetcli-fb (1:2.1.53-1ubuntu3) ...
Setting up python3-wcwidth (0.2.5+dfsg1-1.1ubuntu1) ...
Setting up python3-typing-extensions (4.10.0-1) ...
Setting up python3-pyudev (0.24.0-1) ...
Setting up python3-rtslib-fb (2.1.74-0ubuntu5) ...
Created symlink /etc/systemd/system/multi-user.target.wants/rtslib-fb-targetctl.service → /usr/lib/systemd/system/rtslib-fb-targetctl.service.
Setting up python3-urwid (2.6.10-1) ...
Setting up python3-configshell-fb (1:1.1.28-2.1) ...
/usr/lib/python3/dist-packages/configshell/console.py:39: SyntaxWarning: invalid escape sequence '\['
  _re_ansi_seq = re.compile('(\033\[..?m)')
/usr/lib/python3/dist-packages/configshell/shell.py:116: SyntaxWarning: invalid escape sequence '\+'
  var = Word(alphanums + '?;&*$!#,=_\+/.<>()~@:-%[]')
/usr/lib/python3/dist-packages/configshell/shell.py:118: SyntaxWarning: invalid escape sequence '\-'
  keyword = Word(alphanums + '_\-')
/usr/lib/python3/dist-packages/configshell/shell.py:124: SyntaxWarning: invalid escape sequence '\['
  pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \
/usr/lib/python3/dist-packages/configshell/shell.py:124: SyntaxWarning: invalid escape sequence '\['
  pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \
/usr/lib/python3/dist-packages/configshell/shell.py:131: SyntaxWarning: invalid escape sequence '\|'
  readline.set_completer_delims('\t\n ~!#$^&(){}\|;\'",?')
/usr/lib/python3/dist-packages/configshell_fb/console.py:39: SyntaxWarning: invalid escape sequence '\['
  _re_ansi_seq = re.compile('(\033\[..?m)')
/usr/lib/python3/dist-packages/configshell_fb/shell.py:116: SyntaxWarning: invalid escape sequence '\+'
  var = Word(alphanums + '?;&*$!#,=_\+/.<>()~@:-%[]')
/usr/lib/python3/dist-packages/configshell_fb/shell.py:118: SyntaxWarning: invalid escape sequence '\-'
  keyword = Word(alphanums + '_\-')
/usr/lib/python3/dist-packages/configshell_fb/shell.py:124: SyntaxWarning: invalid escape sequence '\['
  pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \
/usr/lib/python3/dist-packages/configshell_fb/shell.py:124: SyntaxWarning: invalid escape sequence '\['
  pathstd = Regex('([A-Za-z0-9:_.\[\]]|-)*' + '/' + '([A-Za-z0-9:_.\[\]/]|-)*') \
/usr/lib/python3/dist-packages/configshell_fb/shell.py:131: SyntaxWarning: invalid escape sequence '\|'
  readline.set_completer_delims('\t\n ~!#$^&(){}\|;\'",?')
Setting up targetcli-fb (1:2.1.53-1ubuntu3) ...
/usr/lib/python3/dist-packages/targetcli/ui_root.py:161: SyntaxWarning: invalid escape sequence '\s'
  backups = [line for line in prefs.read().splitlines() if re.match('^max_backup_files\s*=', line)]
Processing triggers for man-db (2.12.0-4build2) ...
Scanning processes...                                                                                                                  
Scanning candidates...                                                                                                                 
Scanning linux images...                                                                                                               

Running kernel seems to be up-to-date.

Restarting services...

...

Kasulikud lisamaterjalid

Pärit leheküljelt "https://www.auul.pri.ee/wiki/index.php?title=ISCSI_offload&oldid=1383"