Lehekülje ZFS kasutamine operatsioonisüsteemiga Debian v. 11 lähteteksti vaatamine

===Sissejuhatus===

TODO

===Paigaldamine===

2021 aasta talvel on root-on-zfs debian 11 tekitamiseks üks suhteliselt mugav võimalus

* kasutada skripti aadressilt https://github.com/hn/debian-buster-zfs-root (debian v. 10)
* uuendada süsteem v. 11 peale
* uuendada zfs pool

 # zpool upgrade rpool

hn kasutaja skript töötab, aga tulemus erineb oluliselt https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html juhendis kirjeldatud viisil saadud süsteemist

* fdisk layout
* dataset layout
* /tmp kataloog on no-exec
* ei ole eraldi bpool ja rpool, on kasutusel ainult rpool
* jne

Debian v. 11 zfs omadusel 2021 talvel

* sisaldab zfs v. 2.0 tarkvara
* os intalleris puudub zfs tugi
* zfs ja os vaheline integratsioon puudub (a la ubuntu zsys, zsysctl jms)

===zfs põhine dual boot===

zfs põhise sama pool datasettide abil tehtud dual boot kasutamiseks sobib

* lähtepunktiks on töötav süsteem
* moodustada juurfailisüsteemile vastavast datasetist snapshot ja kloon

 # zfs snapshot rpool/ROOT/debian-buster@ss
 # zfs list -t snapshot
 # zfs clone rpool/ROOT/debian-buster@ss rpool/ROOT/debian-tmp

* grub bootloaderis muuta rpool/ROOT/debian-buster -> rpool/ROOT/debian-tmp (kahes kohas)
* bootida break=mount abil initramfs prompti
* eemaldada käesoleval juurikal mountpoint

 zpool import rpool
 zfs set mountpoint=none rpool/ROOT/debian-buster

* lisada uuelda juurikale mountpoint

 zfs set mountpoint=/ rpool/ROOT/debian-tmp
 zpool export rpool

* kuna /etc/fstab peale pole zfs ühenduskohta kirjeldatud, siis fstab failis muudatusi pole vaja teha
* bootida arvutit edasi, ctrl-d
* klooni jäädavalt kasutama jäädes tuleks teha promote ning eemaldada kloon ja snapshot

 # zfs promote rpool/ROOT/debian-tmp

Tulemusena saab sisse logida, kasutada jne.

===df -h vs zfs list===

zpool list väljund

<pre>
root@deb11-tookoht:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool   480M   153M   327M        -         -    13%    31%  1.00x    ONLINE  -
rpool  62.5G  58.4G  4.07G        -         -    84%    93%  1.00x    ONLINE  -
</pre>

zfs list väljund

<pre>
root@deb11-tookoht:~# zfs list
NAME                USED  AVAIL     REFER  MOUNTPOINT
bpool               152M   200M       96K  /
bpool/BOOT          151M   200M       96K  none
bpool/BOOT/debian   151M   200M      112M  legacy
rpool              59.4G  1.11G       96K  /
rpool/ROOT         31.3G  1.11G       96K  none
rpool/ROOT/debian  31.3G  1.11G     30.7G  /
rpool/home         18.2G  1.11G      584M  /home
rpool/home/imre    17.5G  1.11G     17.5G  /home/imre
rpool/home/root    94.2M  1.11G     94.2M  /root
rpool/swap         4.25G  2.12G     3.24G  -
rpool/var          5.68G  1.11G       96K  /var
rpool/var/cache    5.58G  1.11G     5.58G  /var/cache
rpool/var/lib        96K  1.11G       96K  /var/lib
rpool/var/log       109M  1.11G      109M  legacy
rpool/var/spool     120K  1.11G      120K  legacy
</pre>

kus

* rpool kõigi non-volume tüüpi datasettide jaoks on sama palju vaba ruumi kuna zfs nö olemus mõttes asuvad nad ühe nö juur-dataset all
* rpool/swap on volume tüüpi dataset ja seetõttu eristub
* dataset USED tulba väärtus sisaldab kõigi alla jäävate datasettide väärtuste summat

Esitada ainult volume tüüpi datasetid

<pre>
root@deb11-tookoht:~# zfs list -t volume
NAME         USED  AVAIL     REFER  MOUNTPOINT
rpool/swap  4.25G  2.12G     3.24G  -
</pre>

zfs list -o space väljund

<pre>
root@deb11-tookoht:~# zfs list -o space
NAME               AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
bpool               200M   152M        0B     96K             0B       152M
bpool/BOOT          200M   151M        0B     96K             0B       151M
bpool/BOOT/debian   200M   151M     38.9M    112M             0B         0B
rpool              1.11G  59.4G        0B     96K             0B      59.4G
rpool/ROOT         1.11G  31.3G        0B     96K             0B      31.3G
rpool/ROOT/debian  1.11G  31.3G      596M   30.7G             0B         0B
rpool/home         1.11G  18.2G        0B    584M             0B      17.6G
rpool/home/imre    1.11G  17.5G        0B   17.5G             0B         0B
rpool/home/root    1.11G  94.2M        0B   94.2M             0B         0B
rpool/swap         2.12G  4.25G        0B   3.24G          1.01G         0B
rpool/var          1.11G  5.68G        0B     96K             0B      5.68G
rpool/var/cache    1.11G  5.58G        0B   5.58G             0B         0B
rpool/var/lib      1.11G    96K        0B     96K             0B         0B
rpool/var/log      1.11G   109M        0B    109M             0B         0B
rpool/var/spool    1.11G   120K        0B    120K             0B         0B
</pre>

kus

* lisaks eelmisele 'zfs list' väljundile esitatakse '-o space' suvandi kasutamisel põhjalikum info

df -h väljund

<pre>
root@deb11-tookoht:~# df -h
Filesystem         Size  Used Avail Use% Mounted on
udev               5.9G     0  5.9G   0% /dev
tmpfs              1.2G  1.3M  1.2G   1% /run
rpool/ROOT/debian   32G   31G  1.2G  97% /
tmpfs              5.9G  572M  5.3G  10% /dev/shm
tmpfs              5.0M  4.0K  5.0M   1% /run/lock
tmpfs              4.0M     0  4.0M   0% /sys/fs/cgroup
rpool/var/spool    1.2G  128K  1.2G   1% /var/spool
rpool/var/log      1.3G  109M  1.2G   9% /var/log
bpool/BOOT/debian  312M  113M  200M  37% /boot
/dev/vda2          505M  5.3M  499M   2% /boot/efi
tmpfs              5.9G  715M  5.2G  12% /tmp
rpool/home         1.7G  585M  1.2G  34% /home
rpool/home/root    1.3G   95M  1.2G   8% /root
rpool/var/cache    6.7G  5.6G  1.2G  84% /var/cache
rpool/home/imre     19G   18G  1.2G  95% /home/imre
tmpfs              1.2G   68K  1.2G   1% /run/user/1000
</pre>

kus

* klassikalise df utiliidi väljund ei sobi hästi kõigi zfs detailide esitamiseks, aga ülevaate saab
* Size tulba väärtuseks esitatakse Used + Avail summa (mis on vastavalt zfs list -o space väljundi USEDDS + AVAIL)

===2022 kevadel zfs kasutamine===

Kasutada olevad plokkseadmed

<pre>
# lsscsi -s
[0:0:0:0]    disk    ATA      Samsung SSD 860  2B6Q  /dev/sda   1.00TB
[1:0:0:0]    disk    ATA      Samsung SSD 860  2B6Q  /dev/sdb   1.00TB
[2:0:0:0]    disk    ATA      Samsung SSD 870  1B6Q  /dev/sdc   4.00TB
[3:0:0:0]    disk    ATA      Samsung SSD 870  1B6Q  /dev/sdd   4.00TB
[N:0:4:1]    disk    Samsung SSD 970 EVO Plus 2TB__1            /dev/nvme0n1  2.00TB
[N:1:4:1]    disk    Samsung SSD 970 EVO Plus 2TB__1            /dev/nvme1n1  2.00TB
</pre>

kus

<pre>
# ls -l /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB016*
lrwxrwxrwx 1 root root 9 May 14 10:22 /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K -> ../../sdd
lrwxrwxrwx 1 root root 9 May 14 10:22 /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01694L -> ../../sdc
</pre>

Moodustatakse ssd 870 seadmetest 2-way mirror

 # zpool create -o ashift=13 zp_evo_870 mirror /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01694L
 # zpool set autoexpand=on zp_evo_870
 # zfs set compression=zstd zp_evo_870
 # zfs set mountpoint=none zp_evo_870

kus

* tark on kasutada füüsiliste seadmetele viitamiseks unikaalseid nimesid (alternatiiv oleks /dev/sdc ja /dev/sdd aga need väärtused sõltuvad ketaste sata portidesse kinnitamise järjekorrast jms)
* väidetavalt on samsung ssd ketaste puhul tark kasutada ashift=13 (ehk 8K plokk) väärtust - https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/

===ZFS failisüsteem on liiga täis ja läks katki===

Peale küllaltki täis failisüsteemi otsas rekursiivse snapshot ütlemist

 # zfs snapshot -r sn_zfs_ssd@replica20220515

ütleb failisüsteemi vastu nii

<pre>
# zfs destroy sn_zfs_ssd/vm-1125-disk-0@replica20220515
internal error: cannot destroy snapshots: Channel number out of range
Aborted
</pre>

kusjuures lisaks sellele, et ta on täis

<pre>
# zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
sn_zfs_ssd  1.81T   953G   903G        -         -    44%    51%  1.00x  DEGRADED  -

# zfs list sn_zfs_ssd -o space
NAME        AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
sn_zfs_ssd     0B  2.14T        0B     96K             0B      2.14T
</pre>

on ta veel mirror-0 ja on katki ka

<pre>
# zpool status -v
  pool: sn_zfs_ssd
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:22:56 with 0 errors on Sun Feb 13 00:46:58 2022
config:

        NAME                      STATE     READ WRITE CKSUM
        sn_zfs_ssd                DEGRADED     0     0     0
          wwn-0x5002538e40a0eb32  DEGRADED     3     0    37  too many errors
          wwn-0x5002538e40a05fa2  DEGRADED     3     2 1.39K  too many errors

errors: Permanent errors have been detected in the following files:

        sn_zfs_ssd/vm-102-disk-5:<0x1>
</pre>

Tundub, et selles olukorras võib olla abi sellisest protsetuurist

* tekitada vabale meediale, nt lvm volume peale vastavatest scsi seadmetest dd abil plokkseadme tasemel koopiad

 # dd if=/dev/sda of=/mnt/sdd/evo-860-sda.img bs=1M status=progress
 # dd if=/dev/sdb of=/mnt/sdd/evo-860-sdb.img bs=1M status=progress

* tekitada proxmox arvutit (nt virtuaalne) ja võtta seal külge koopiaid sisaldav failisüsteem ja lisaks veel suurem muu failisüsteem
* moodustada virtuaalne arvuti ja teha talle kõik seadmed (kokku neli) nähtavaks
* teha zpool import

 # zpool import
 ..

* teha ühe kaupa zpool seadme asendus suurema vastu (toimub resilvering)

<pre>
# zpool replace sn_zfs_ssd /dev/vdb /dev/vdd
# zpool replace sn_zfs_ssd /dev/vdc /dev/vde

# zpool status
  pool: sn_zfs_ssd
 state: ONLINE
  scan: resilvered 445G in 00:44:11 with 0 errors on Sun May 15 23:06:07 2022
config:

	NAME        STATE     READ WRITE CKSUM
	sn_zfs_ssd  ONLINE       0     0     0
	  vdd       ONLINE       0     0     0
	  vde       ONLINE       0     0     0
<pre>

* paistab, et süsteem on oluliselt paranenud, nüüd saab öelda edasi zfs online -e sn_zfs_ssd /dev/vdd ja zpool online -e sn_zfs_ssd /dev/vde

Tulemusena on olukord jällegi kontrolli all, zpool ja zfs käsud töötavad jne (nt saab snapshotisid tekitada ja eemaldada jne)

<pre>
# zpool list sn_zfs_ssd
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
sn_zfs_ssd  3.11T   953G  2.18T        -         -    25%    29%  1.00x    ONLINE  -
# zfs list sn_zfs_ssd
NAME         USED  AVAIL     REFER  MOUNTPOINT
sn_zfs_ssd  1.52T  1.49T       96K  /sn_zfs_ssd
</pre>

parandatud tõmmised saab järgmises arvutis kasutusse võtta nt qemu-nbd utiliidi abil

 # qemu-nbd -c /dev/nbd0 /mnt/sdd-ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K/dir/images/102/vm-102-disk-0.qcow2
 # qemu-nbd -c /dev/nbd1 /mnt/sdd-ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K/dir/images/102/vm-102-disk-1.qcow2

Seejärel zpool import peaks avastama ressursi. Kasutamise lõpus

 # zpool export poolnimi
 # qemu-nbd -d /dev/nbd0
 # qemu-nbd -d /dev/nbd1

Raid-0 pool kasutamise jätkamiseks raid-1 poolina sobib öelda

 # zpool attach sn_zfs_ssd /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01694L /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K

Tulemusena algab resilverdamine

<pre>
# zpool status
  pool: sn_zfs_ssd
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed May 18 18:11:05 2022
	222G scanned at 1.98G/s, 29.1G issued at 266M/s, 1.58T total
	29.1G resilvered, 1.79% done, 01:42:17 to go
config:

	NAME                                             STATE     READ WRITE CKSUM
	sn_zfs_ssd                                       ONLINE       0     0     0
	  mirror-0                                       ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01694L  ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K  ONLINE       0     0     0  (resilvering)

errors: No known data errors
</pre>

Kasulikud lisamaterjalid

* https://github.com/openzfs/zfs/issues/9849

===zpool sisu kopeerimine teisse zpool'i===

Esmalt moodustatakse lähte poolis snapshot, tulemusena tekib kõigile datasettidele v.a. olemasolevad snapshotid jms, snapshot, sn_zfs_ssd on pool nimi; -r on rekursiivselt

 # zfs snapshot -r sn_zfs_ssd@replica20220516

Seejärel kopeeritakse, zp_evo_870 on sihtkohaks oleva pool nimi

 # zfs send -R sn_zfs_ssd@replica20220516 | zfs receive -udF zp_evo_870

kus

* -R - tekitada nn replication stream, see sisaldab nö kõike näidatud snapshotile eelnevat
* -u - sihtpunktis mitte püüda monteerida
* -F - kui kopeeritav dataset on olemas, kustutada
* -d - xxx

Tulemusena tekib destination pool peale see snaphost ja seejuures ka kõik nö mis sellest ettepoole jääb (dataset'id, varasemad snapshotid jms). Tõenäoliselt on asjakohane lähte poolist ja siht poolist nö kopeerimise-snapshot eemaldada

 # zfs destroy -r sn_zfs_ssd@replica20220516
 # zfs destroy -r zp_evo_870@replica20220516

Tundub, et kopeerimise ajal võib tekkida sarnaseid veateateid ja ehk see ei ole nö maailmalõpp

<pre>
[Mon May 16 02:14:22 2022] block nbd1: Possible stuck request 0000000025b58da6: control (read@509541847040,131072B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd0: Possible stuck request 00000000db090ca2: control (read@458513432576,126976B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd1: Possible stuck request 00000000b3423ad4: control (read@509541978112,131072B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd1: Possible stuck request 00000000e61e4d21: control (read@509542109184,131072B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd0: Possible stuck request 00000000dcbd73bf: control (read@458513690624,131072B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd0: Possible stuck request 000000001aa586ea: control (read@458513821696,106496B). Runtime 30 seconds
[Mon May 16 02:14:33 2022] ata1.00: exception Emask 0x0 SAct 0x26ec1000 SErr 0x0 action 0x6 frozen
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/00:60:d8:dd:d5/01:00:ac:01:00/40 tag 12 ncq dma 131072 in
                                    res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/d0:90:d8:de:d5/00:00:ac:01:00/40 tag 18 ncq dma 106496 in
                                    res 40/00:01:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/e0:98:20:00:00/00:00:00:00:00/40 tag 19 ncq dma 114688 in
                                    res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/b8:a8:20:dc:d5/00:00:ac:01:00/40 tag 21 ncq dma 94208 in
                                    res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/00:b0:48:5c:04/01:00:e1:00:00/40 tag 22 ncq dma 131072 in
                                    res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/e0:b8:20:00:00/00:00:00:00:00/40 tag 23 ncq dma 114688 in
                                    res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/00:c8:48:5d:04/01:00:e1:00:00/40 tag 25 ncq dma 131072 in
                                    res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/e0:d0:20:00:00/00:00:00:00:00/40 tag 26 ncq dma 114688 in
                                    res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/e0:e8:20:00:00/00:00:00:00:00/40 tag 29 ncq dma 114688 in
                                    res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1: hard resetting link
[Mon May 16 02:14:39 2022] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Mon May 16 02:14:39 2022] ata1.00: supports DRM functions and may not be fully accessible
[Mon May 16 02:14:39 2022] ata1.00: disabling queued TRIM support
[Mon May 16 02:14:39 2022] ata1.00: supports DRM functions and may not be fully accessible
[Mon May 16 02:14:39 2022] ata1.00: disabling queued TRIM support
[Mon May 16 02:14:39 2022] ata1.00: configured for UDMA/133
[Mon May 16 02:14:39 2022] ata1.00: device reported invalid CHS sector 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#12 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#12 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#12 CDB: Read(16) 88 00 00 00 00 01 ac d5 dd d8 00 00 01 00 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 7194664408 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#18 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#18 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#18 CDB: Read(16) 88 00 00 00 00 01 ac d5 de d8 00 00 00 d0 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 7194664664 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#21 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#21 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#21 CDB: Read(16) 88 00 00 00 00 01 ac d5 dc 20 00 00 00 b8 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 7194663968 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#22 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#22 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#22 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#22 CDB: Read(16) 88 00 00 00 00 00 e1 04 5c 48 00 00 01 00 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 3775159368 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#25 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#25 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#25 CDB: Read(16) 88 00 00 00 00 00 e1 04 5d 48 00 00 01 00 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 3775159624 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 0
[Mon May 16 02:14:39 2022] ata1: EH complete
[Mon May 16 02:14:39 2022] ata1.00: Enabling discard_zeroes_data
</pre>

===Misc===

suuruste arvutamine

<pre>
# a=0; poolname="sn_zfs_ssd"; for i in `zfs list -H -r $poolname -o name,used -p | grep "^$poolname/" | awk {'print $2'}`; do a=`echo $a + $i | bc`; done; sumg=`echo $a/1024^3 | bc`; echo $sumg
1645

# a=0; poolname="zp_evo_870"; for i in `zfs list -H -r $poolname -o name,used -p | grep "^$poolname/" | awk {'print $2'}`; do a=`echo $a + $i | bc`; done; sumg=`echo $a/1024^3 | bc`; echo $sumg
783
</pre>

kopeerimine, variant 1

 # for i in `zfs list -t volume -r sn_zfs_ssd | grep ^sn | awk {'print $1'} | cut -d "/" -f 2`; do echo $i; zfs send sn_zfs_ssd/$i | zfs receive zp_evo_870/$i; echo $?; done

kopeerimine, variant 2

===zvol moodustamine ja kasutamine===

zvol on zfs nö eripärane dataset ressurss peale moodustatakse reeglina mingi muu andmete haldamise kiht, nt lvm, failisüsteem vms (või jagatakse välja üle võrgu iscsi abil). zvol tekitamiseks sobib öelda

 # zfs create -V 10G zpool_wdc/iscsi_pbs

zvol peale ext4 failisüsteemi tekitamiseks sobib öelda

 # mkfs.ext4 /dev/zpool_wdc/iscsi_pbs

zvol ressursi suurendamiseks sobib öelda

 # zfs set volsize=30G zpool_wdc/iscsi_pbs

zvol ressursi kustutamiseks sobib öelda

 # zfs destroy zpool_wdc/iscsi_pbs

===zvol suuruste arvutamine===

Olgu paista selline ressursside kasutus

<pre>
root@pm60-trt:~# zfs list -o space | head -n 8
NAME                                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
sn_zfs_ssd                                  186G  3.33T        0B    192K             0B      3.33T
sn_zfs_ssd/imretestvolume                   194G  8.25G        0B    112K          8.25G         0B
sn_zfs_ssd/subvol-161-disk-0               7.18G   835M        0B    835M             0B         0B
sn_zfs_ssd/vm-100-disk-0                    213G  82.5G        0B   55.1G          27.4G         0B
sn_zfs_ssd/vm-100-disk-1                    186G     3M        0B    176K          2.83M         0B
sn_zfs_ssd/vm-102-disk-5                    187G  22.7G        0B   21.6G          1.11G         0B
sn_zfs_ssd/vm-1033-disk-0                   187G  12.4G        0B   11.9G           525M         0B
</pre>

kusjuures nt

<pre>
root@pm60-trt:~# zfs get all sn_zfs_ssd/vm-100-disk-0 | egrep " used | available | referenced | volsize | refreservation | usedbydataset | usedbyrefreservation"
sn_zfs_ssd/vm-100-disk-0  used                  82.5G                  -
sn_zfs_ssd/vm-100-disk-0  available             213G                   -
sn_zfs_ssd/vm-100-disk-0  referenced            55.1G                  -
sn_zfs_ssd/vm-100-disk-0  volsize               80G                    local
sn_zfs_ssd/vm-100-disk-0  refreservation        82.5G                  local
sn_zfs_ssd/vm-100-disk-0  usedbydataset         55.1G                  -
sn_zfs_ssd/vm-100-disk-0  usedbyrefreservation  27.4G                  -
</pre>

Tulemust sobib tõlgendada selliselt

* used ja refreservation (ja vist ka volsize) väljendavad proxmox webgui liideses vmid=100 arvuti kettale näidatud mahtu
* referenced ja usedbydataset näitavad kui palju reaalselt virtuaalne arvuti kasutab
* usedbyrefreservation näitab kui palju on nö zpool ressursist kinni täna refreservationile, st 82.5 - 55.1 = 27.4
* available näitab kui palju on zpool peal reaalselt kasutada arvestades sinna hulla refreservation all kinni oleva mahu, 186 + 27.4 = 213.4

===Kasulikud lisamaterjalid===

* TODO