ZFS kasutamine operatsioonisüsteemiga Debian v. 11

Sissejuhatus

TODO

Paigaldamine

2021 aasta talvel on root-on-zfs debian 11 tekitamiseks üks suhteliselt mugav võimalus

kasutada skripti aadressilt https://github.com/hn/debian-buster-zfs-root (debian v. 10)
uuendada süsteem v. 11 peale
uuendada zfs pool

# zpool upgrade rpool

hn kasutaja skript töötab, aga tulemus erineb oluliselt https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html juhendis kirjeldatud viisil saadud süsteemist

fdisk layout
dataset layout
/tmp kataloog on no-exec
ei ole eraldi bpool ja rpool, on kasutusel ainult rpool
jne

Debian v. 11 zfs omadusel 2021 talvel

sisaldab zfs v. 2.0 tarkvara
os intalleris puudub zfs tugi
zfs ja os vaheline integratsioon puudub (a la ubuntu zsys, zsysctl jms)

zfs põhine dual boot

zfs põhise sama pool datasettide abil tehtud dual boot kasutamiseks sobib

lähtepunktiks on töötav süsteem
moodustada juurfailisüsteemile vastavast datasetist snapshot ja kloon

# zfs snapshot rpool/ROOT/debian-buster@ss
# zfs list -t snapshot
# zfs clone rpool/ROOT/debian-buster@ss rpool/ROOT/debian-tmp

grub bootloaderis muuta rpool/ROOT/debian-buster -> rpool/ROOT/debian-tmp (kahes kohas)
bootida break=mount abil initramfs prompti
eemaldada käesoleval juurikal mountpoint

zpool import rpool
zfs set mountpoint=none rpool/ROOT/debian-buster

lisada uuelda juurikale mountpoint

zfs set mountpoint=/ rpool/ROOT/debian-tmp
zpool export rpool

kuna /etc/fstab peale pole zfs ühenduskohta kirjeldatud, siis fstab failis muudatusi pole vaja teha
bootida arvutit edasi, ctrl-d
klooni jäädavalt kasutama jäädes tuleks teha promote ning eemaldada kloon ja snapshot

# zfs promote rpool/ROOT/debian-tmp

Tulemusena saab sisse logida, kasutada jne.

df -h vs zfs list

zpool list väljund

root@deb11-tookoht:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool   480M   153M   327M        -         -    13%    31%  1.00x    ONLINE  -
rpool  62.5G  58.4G  4.07G        -         -    84%    93%  1.00x    ONLINE  -

zfs list väljund

root@deb11-tookoht:~# zfs list
NAME                USED  AVAIL     REFER  MOUNTPOINT
bpool               152M   200M       96K  /
bpool/BOOT          151M   200M       96K  none
bpool/BOOT/debian   151M   200M      112M  legacy
rpool              59.4G  1.11G       96K  /
rpool/ROOT         31.3G  1.11G       96K  none
rpool/ROOT/debian  31.3G  1.11G     30.7G  /
rpool/home         18.2G  1.11G      584M  /home
rpool/home/imre    17.5G  1.11G     17.5G  /home/imre
rpool/home/root    94.2M  1.11G     94.2M  /root
rpool/swap         4.25G  2.12G     3.24G  -
rpool/var          5.68G  1.11G       96K  /var
rpool/var/cache    5.58G  1.11G     5.58G  /var/cache
rpool/var/lib        96K  1.11G       96K  /var/lib
rpool/var/log       109M  1.11G      109M  legacy
rpool/var/spool     120K  1.11G      120K  legacy

kus

rpool kõigi non-volume tüüpi datasettide jaoks on sama palju vaba ruumi kuna zfs nö olemus mõttes asuvad nad ühe nö juur-dataset all
rpool/swap on volume tüüpi dataset ja seetõttu eristub
dataset USED tulba väärtus sisaldab kõigi alla jäävate datasettide väärtuste summat

Esitada ainult volume tüüpi datasetid

root@deb11-tookoht:~# zfs list -t volume
NAME         USED  AVAIL     REFER  MOUNTPOINT
rpool/swap  4.25G  2.12G     3.24G  -

zfs list -o space väljund

root@deb11-tookoht:~# zfs list -o space
NAME               AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
bpool               200M   152M        0B     96K             0B       152M
bpool/BOOT          200M   151M        0B     96K             0B       151M
bpool/BOOT/debian   200M   151M     38.9M    112M             0B         0B
rpool              1.11G  59.4G        0B     96K             0B      59.4G
rpool/ROOT         1.11G  31.3G        0B     96K             0B      31.3G
rpool/ROOT/debian  1.11G  31.3G      596M   30.7G             0B         0B
rpool/home         1.11G  18.2G        0B    584M             0B      17.6G
rpool/home/imre    1.11G  17.5G        0B   17.5G             0B         0B
rpool/home/root    1.11G  94.2M        0B   94.2M             0B         0B
rpool/swap         2.12G  4.25G        0B   3.24G          1.01G         0B
rpool/var          1.11G  5.68G        0B     96K             0B      5.68G
rpool/var/cache    1.11G  5.58G        0B   5.58G             0B         0B
rpool/var/lib      1.11G    96K        0B     96K             0B         0B
rpool/var/log      1.11G   109M        0B    109M             0B         0B
rpool/var/spool    1.11G   120K        0B    120K             0B         0B

kus

lisaks eelmisele 'zfs list' väljundile esitatakse '-o space' suvandi kasutamisel põhjalikum info

df -h väljund

root@deb11-tookoht:~# df -h
Filesystem         Size  Used Avail Use% Mounted on
udev               5.9G     0  5.9G   0% /dev
tmpfs              1.2G  1.3M  1.2G   1% /run
rpool/ROOT/debian   32G   31G  1.2G  97% /
tmpfs              5.9G  572M  5.3G  10% /dev/shm
tmpfs              5.0M  4.0K  5.0M   1% /run/lock
tmpfs              4.0M     0  4.0M   0% /sys/fs/cgroup
rpool/var/spool    1.2G  128K  1.2G   1% /var/spool
rpool/var/log      1.3G  109M  1.2G   9% /var/log
bpool/BOOT/debian  312M  113M  200M  37% /boot
/dev/vda2          505M  5.3M  499M   2% /boot/efi
tmpfs              5.9G  715M  5.2G  12% /tmp
rpool/home         1.7G  585M  1.2G  34% /home
rpool/home/root    1.3G   95M  1.2G   8% /root
rpool/var/cache    6.7G  5.6G  1.2G  84% /var/cache
rpool/home/imre     19G   18G  1.2G  95% /home/imre
tmpfs              1.2G   68K  1.2G   1% /run/user/1000

kus

klassikalise df utiliidi väljund ei sobi hästi kõigi zfs detailide esitamiseks, aga ülevaate saab
Size tulba väärtuseks esitatakse Used + Avail summa (mis on vastavalt zfs list -o space väljundi USEDDS + AVAIL)

2022 kevadel zfs kasutamine

Kasutada olevad plokkseadmed

# lsscsi -s
[0:0:0:0]    disk    ATA      Samsung SSD 860  2B6Q  /dev/sda   1.00TB
[1:0:0:0]    disk    ATA      Samsung SSD 860  2B6Q  /dev/sdb   1.00TB
[2:0:0:0]    disk    ATA      Samsung SSD 870  1B6Q  /dev/sdc   4.00TB
[3:0:0:0]    disk    ATA      Samsung SSD 870  1B6Q  /dev/sdd   4.00TB
[N:0:4:1]    disk    Samsung SSD 970 EVO Plus 2TB__1            /dev/nvme0n1  2.00TB
[N:1:4:1]    disk    Samsung SSD 970 EVO Plus 2TB__1            /dev/nvme1n1  2.00TB

kus

# ls -l /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB016*
lrwxrwxrwx 1 root root 9 May 14 10:22 /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K -> ../../sdd
lrwxrwxrwx 1 root root 9 May 14 10:22 /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01694L -> ../../sdc

Moodustatakse ssd 870 seadmetest 2-way mirror

# zpool create -o ashift=13 zp_evo_870 mirror /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01694L
# zpool set autoexpand=on zp_evo_870
# zfs set compression=zstd zp_evo_870
# zfs set mountpoint=none zp_evo_870

kus

tark on kasutada füüsiliste seadmetele viitamiseks unikaalseid nimesid (alternatiiv oleks /dev/sdc ja /dev/sdd aga need väärtused sõltuvad ketaste sata portidesse kinnitamise järjekorrast jms)
väidetavalt on samsung ssd ketaste puhul tark kasutada ashift=13 (ehk 8K plokk) väärtust - https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/

ZFS failisüsteem on liiga täis ja läks katki

Peale küllaltki täis failisüsteemi otsas rekursiivse snapshot ütlemist

# zfs snapshot -r sn_zfs_ssd@replica20220515

ütleb failisüsteemi vastu nii

# zfs destroy sn_zfs_ssd/vm-1125-disk-0@replica20220515
internal error: cannot destroy snapshots: Channel number out of range
Aborted

kusjuures lisaks sellele, et ta on täis

# zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
sn_zfs_ssd  1.81T   953G   903G        -         -    44%    51%  1.00x  DEGRADED  -

# zfs list sn_zfs_ssd -o space
NAME        AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
sn_zfs_ssd     0B  2.14T        0B     96K             0B      2.14T

on ta veel mirror-0 ja on katki ka

# zpool status -v
  pool: sn_zfs_ssd
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:22:56 with 0 errors on Sun Feb 13 00:46:58 2022
config:

        NAME                      STATE     READ WRITE CKSUM
        sn_zfs_ssd                DEGRADED     0     0     0
          wwn-0x5002538e40a0eb32  DEGRADED     3     0    37  too many errors
          wwn-0x5002538e40a05fa2  DEGRADED     3     2 1.39K  too many errors

errors: Permanent errors have been detected in the following files:

        sn_zfs_ssd/vm-102-disk-5:<0x1>

Tundub, et selles olukorras võib olla abi sellisest protsetuurist

tekitada vabale meediale, nt lvm volume peale vastavatest scsi seadmetest dd abil plokkseadme tasemel koopiad

# dd if=/dev/sda of=/mnt/sdd/evo-860-sda.img bs=1M status=progress
# dd if=/dev/sdb of=/mnt/sdd/evo-860-sdb.img bs=1M status=progress

tekitada proxmox arvutit (nt virtuaalne) ja võtta seal külge koopiaid sisaldav failisüsteem ja lisaks veel suurem muu failisüsteem
moodustada virtuaalne arvuti ja teha talle kõik seadmed (kokku neli) nähtavaks
teha zpool import

# zpool import
..

teha ühe kaupa zpool seadme asendus suurema vastu (toimub resilvering)

# zpool replace sn_zfs_ssd /dev/vdb /dev/vdd
# zpool replace sn_zfs_ssd /dev/vdc /dev/vde

# zpool status
  pool: sn_zfs_ssd
 state: ONLINE
  scan: resilvered 445G in 00:44:11 with 0 errors on Sun May 15 23:06:07 2022
config:

	NAME        STATE     READ WRITE CKSUM
	sn_zfs_ssd  ONLINE       0     0     0
	  vdd       ONLINE       0     0     0
	  vde       ONLINE       0     0     0
<pre>

* paistab, et süsteem on oluliselt paranenud, nüüd saab öelda edasi zfs online -e sn_zfs_ssd /dev/vdd ja zpool online -e sn_zfs_ssd /dev/vde

Tulemusena on olukord jällegi kontrolli all, zpool ja zfs käsud töötavad jne (nt saab snapshotisid tekitada ja eemaldada jne)

<pre>
# zpool list sn_zfs_ssd
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
sn_zfs_ssd  3.11T   953G  2.18T        -         -    25%    29%  1.00x    ONLINE  -
# zfs list sn_zfs_ssd
NAME         USED  AVAIL     REFER  MOUNTPOINT
sn_zfs_ssd  1.52T  1.49T       96K  /sn_zfs_ssd

parandatud tõmmised saab järgmises arvutis kasutusse võtta nt qemu-nbd utiliidi abil

# qemu-nbd -c /dev/nbd0 /mnt/sdd-ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K/dir/images/102/vm-102-disk-0.qcow2
# qemu-nbd -c /dev/nbd1 /mnt/sdd-ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K/dir/images/102/vm-102-disk-1.qcow2

Seejärel zpool import peaks avastama ressursi. Kasutamise lõpus

# zpool export poolnimi
# qemu-nbd -d /dev/nbd0
# qemu-nbd -d /dev/nbd1

Raid-0 pool kasutamise jätkamiseks raid-1 poolina sobib öelda

# zpool attach sn_zfs_ssd /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01694L /dev/disk/by-id/ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K

Tulemusena algab resilverdamine

# zpool status
  pool: sn_zfs_ssd
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed May 18 18:11:05 2022
	222G scanned at 1.98G/s, 29.1G issued at 266M/s, 1.58T total
	29.1G resilvered, 1.79% done, 01:42:17 to go
config:

	NAME                                             STATE     READ WRITE CKSUM
	sn_zfs_ssd                                       ONLINE       0     0     0
	  mirror-0                                       ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01694L  ONLINE       0     0     0
	    ata-Samsung_SSD_870_EVO_4TB_S6BCNS0RB01688K  ONLINE       0     0     0  (resilvering)

errors: No known data errors

Kasulikud lisamaterjalid

https://github.com/openzfs/zfs/issues/9849

zpool sisu kopeerimine teisse zpool'i

Esmalt moodustatakse lähte poolis snapshot, tulemusena tekib kõigile datasettidele v.a. olemasolevad snapshotid jms, snapshot, sn_zfs_ssd on pool nimi; -r on rekursiivselt

# zfs snapshot -r sn_zfs_ssd@replica20220516

Seejärel kopeeritakse, zp_evo_870 on sihtkohaks oleva pool nimi

# zfs send -R sn_zfs_ssd@replica20220516 | zfs receive -udF zp_evo_870

kus

-R - tekitada nn replication stream, see sisaldab nö kõike näidatud snapshotile eelnevat
-u - sihtpunktis mitte püüda monteerida
-F - kui kopeeritav dataset on olemas, kustutada
-d - xxx

Tulemusena tekib destination pool peale see snaphost ja seejuures ka kõik nö mis sellest ettepoole jääb (dataset'id, varasemad snapshotid jms). Tõenäoliselt on asjakohane lähte poolist ja siht poolist nö kopeerimise-snapshot eemaldada

# zfs destroy -r sn_zfs_ssd@replica20220516
# zfs destroy -r zp_evo_870@replica20220516

Tundub, et kopeerimise ajal võib tekkida sarnaseid veateateid ja ehk see ei ole nö maailmalõpp

[Mon May 16 02:14:22 2022] block nbd1: Possible stuck request 0000000025b58da6: control (read@509541847040,131072B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd0: Possible stuck request 00000000db090ca2: control (read@458513432576,126976B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd1: Possible stuck request 00000000b3423ad4: control (read@509541978112,131072B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd1: Possible stuck request 00000000e61e4d21: control (read@509542109184,131072B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd0: Possible stuck request 00000000dcbd73bf: control (read@458513690624,131072B). Runtime 30 seconds
[Mon May 16 02:14:22 2022] block nbd0: Possible stuck request 000000001aa586ea: control (read@458513821696,106496B). Runtime 30 seconds
[Mon May 16 02:14:33 2022] ata1.00: exception Emask 0x0 SAct 0x26ec1000 SErr 0x0 action 0x6 frozen
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/00:60:d8:dd:d5/01:00:ac:01:00/40 tag 12 ncq dma 131072 in
                                    res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/d0:90:d8:de:d5/00:00:ac:01:00/40 tag 18 ncq dma 106496 in
                                    res 40/00:01:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/e0:98:20:00:00/00:00:00:00:00/40 tag 19 ncq dma 114688 in
                                    res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/b8:a8:20:dc:d5/00:00:ac:01:00/40 tag 21 ncq dma 94208 in
                                    res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/00:b0:48:5c:04/01:00:e1:00:00/40 tag 22 ncq dma 131072 in
                                    res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/e0:b8:20:00:00/00:00:00:00:00/40 tag 23 ncq dma 114688 in
                                    res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/00:c8:48:5d:04/01:00:e1:00:00/40 tag 25 ncq dma 131072 in
                                    res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/e0:d0:20:00:00/00:00:00:00:00/40 tag 26 ncq dma 114688 in
                                    res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1.00: failed command: READ FPDMA QUEUED
[Mon May 16 02:14:33 2022] ata1.00: cmd 60/e0:e8:20:00:00/00:00:00:00:00/40 tag 29 ncq dma 114688 in
                                    res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[Mon May 16 02:14:33 2022] ata1.00: status: { DRDY }
[Mon May 16 02:14:33 2022] ata1: hard resetting link
[Mon May 16 02:14:39 2022] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Mon May 16 02:14:39 2022] ata1.00: supports DRM functions and may not be fully accessible
[Mon May 16 02:14:39 2022] ata1.00: disabling queued TRIM support
[Mon May 16 02:14:39 2022] ata1.00: supports DRM functions and may not be fully accessible
[Mon May 16 02:14:39 2022] ata1.00: disabling queued TRIM support
[Mon May 16 02:14:39 2022] ata1.00: configured for UDMA/133
[Mon May 16 02:14:39 2022] ata1.00: device reported invalid CHS sector 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#12 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#12 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#12 CDB: Read(16) 88 00 00 00 00 01 ac d5 dd d8 00 00 01 00 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 7194664408 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#18 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#18 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#18 CDB: Read(16) 88 00 00 00 00 01 ac d5 de d8 00 00 00 d0 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 7194664664 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#21 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#21 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#21 CDB: Read(16) 88 00 00 00 00 01 ac d5 dc 20 00 00 00 b8 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 7194663968 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#22 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#22 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#22 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#22 CDB: Read(16) 88 00 00 00 00 00 e1 04 5c 48 00 00 01 00 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 3775159368 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 0
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=46s
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#25 Sense Key : Illegal Request [current] 
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#25 Add. Sense: Unaligned write command
[Mon May 16 02:14:39 2022] sd 0:0:0:0: [sda] tag#25 CDB: Read(16) 88 00 00 00 00 00 e1 04 5d 48 00 00 01 00 00 00
[Mon May 16 02:14:39 2022] blk_update_request: I/O error, dev sda, sector 3775159624 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 0
[Mon May 16 02:14:39 2022] ata1: EH complete
[Mon May 16 02:14:39 2022] ata1.00: Enabling discard_zeroes_data

Misc

suuruste arvutamine

# a=0; poolname="sn_zfs_ssd"; for i in `zfs list -H -r $poolname -o name,used -p | grep "^$poolname/" | awk {'print $2'}`; do a=`echo $a + $i | bc`; done; sumg=`echo $a/1024^3 | bc`; echo $sumg
1645

# a=0; poolname="zp_evo_870"; for i in `zfs list -H -r $poolname -o name,used -p | grep "^$poolname/" | awk {'print $2'}`; do a=`echo $a + $i | bc`; done; sumg=`echo $a/1024^3 | bc`; echo $sumg
783

kopeerimine, variant 1

# for i in `zfs list -t volume -r sn_zfs_ssd | grep ^sn | awk {'print $1'} | cut -d "/" -f 2`; do echo $i; zfs send sn_zfs_ssd/$i | zfs receive zp_evo_870/$i; echo $?; done

kopeerimine, variant 2

zvol moodustamine ja kasutamine

zvol on zfs nö eripärane dataset ressurss peale moodustatakse reeglina mingi muu andmete haldamise kiht, nt lvm, failisüsteem vms (või jagatakse välja üle võrgu iscsi abil). zvol tekitamiseks sobib öelda

# zfs create -V 10G zpool_wdc/iscsi_pbs

zvol peale ext4 failisüsteemi tekitamiseks sobib öelda

# mkfs.ext4 /dev/zpool_wdc/iscsi_pbs

zvol ressursi suurendamiseks sobib öelda

# zfs set volsize=30G zpool_wdc/iscsi_pbs

zvol ressursi kustutamiseks sobib öelda

# zfs destroy zpool_wdc/iscsi_pbs

zvol suuruste arvutamine

Olgu paista selline ressursside kasutus

root@pm60-trt:~# zfs list -o space | head -n 8
NAME                                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
sn_zfs_ssd                                  186G  3.33T        0B    192K             0B      3.33T
sn_zfs_ssd/imretestvolume                   194G  8.25G        0B    112K          8.25G         0B
sn_zfs_ssd/subvol-161-disk-0               7.18G   835M        0B    835M             0B         0B
sn_zfs_ssd/vm-100-disk-0                    213G  82.5G        0B   55.1G          27.4G         0B
sn_zfs_ssd/vm-100-disk-1                    186G     3M        0B    176K          2.83M         0B
sn_zfs_ssd/vm-102-disk-5                    187G  22.7G        0B   21.6G          1.11G         0B
sn_zfs_ssd/vm-1033-disk-0                   187G  12.4G        0B   11.9G           525M         0B

kusjuures nt

root@pm60-trt:~# zfs get all sn_zfs_ssd/vm-100-disk-0 | egrep " used | available | referenced | volsize | refreservation | usedbydataset | usedbyrefreservation"
sn_zfs_ssd/vm-100-disk-0  used                  82.5G                  -
sn_zfs_ssd/vm-100-disk-0  available             213G                   -
sn_zfs_ssd/vm-100-disk-0  referenced            55.1G                  -
sn_zfs_ssd/vm-100-disk-0  volsize               80G                    local
sn_zfs_ssd/vm-100-disk-0  refreservation        82.5G                  local
sn_zfs_ssd/vm-100-disk-0  usedbydataset         55.1G                  -
sn_zfs_ssd/vm-100-disk-0  usedbyrefreservation  27.4G                  -

Tulemust sobib tõlgendada selliselt

used ja refreservation (ja vist ka volsize) väljendavad proxmox webgui liideses vmid=100 arvuti kettale näidatud mahtu
referenced ja usedbydataset näitavad kui palju reaalselt virtuaalne arvuti kasutab
usedbyrefreservation näitab kui palju on nö zpool ressursist kinni täna refreservationile, st 82.5 - 55.1 = 27.4
available näitab kui palju on zpool peal reaalselt kasutada arvestades sinna hulla refreservation all kinni oleva mahu, 186 + 27.4 = 213.4

2022 sügis Promxox backup server 2.2

Tunnuseks on zfs failisüsteemi hangumine find /kataloog väljundis jms

root@pbs:~# zpool status
  pool: zpool_wdc
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub in progress since Sun Oct  9 00:24:01 2022
        21.3T scanned at 121M/s, 21.1T issued at 120M/s, 30.2T total
        404K repaired, 69.91% done, 22:00:01 to go
config:

        NAME                                      STATE     READ WRITE CKSUM
        zpool_wdc                                 ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            scsi-0QEMU_QEMU_HARDDISK_drive-scsi0  ONLINE       0     0     3  (repairing)
            scsi-0QEMU_QEMU_HARDDISK_drive-scsi2  ONLINE       0     0     4  (repairing)
            scsi-0QEMU_QEMU_HARDDISK_drive-scsi1  ONLINE       0     0     3  (repairing)
            scsi-0QEMU_QEMU_HARDDISK_drive-scsi3  ONLINE       0     0     1  (repairing)

errors: No known data errors

Syslog sisuldab seejuures

Oct  9 11:45:52 pbs kernel: [204607.143673] general protection fault, probably for non-canonical address 0xdead000000000108: 0000 [#1] SMP NOPTI
Oct  9 11:45:52 pbs kernel: [204607.189034] CPU: 0 PID: 598 Comm: z_rd_int Tainted: P           O      5.15.60-1-pve #1
Oct  9 11:45:52 pbs kernel: [204607.190075] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Oct  9 11:45:52 pbs kernel: [204607.190886] RIP: 0010:abd_free+0xd1/0x1e0 [zfs]
Oct  9 11:45:52 pbs kernel: [204607.200916] Code: 6b 38 4c 89 f7 e8 df 21 35 ec 4c 89 7b 40 49 8b 44 24 50 4c 89 ef 48 be 00 01 00 00 00 00 ad de 48 01 d8 48 8b 08 48 8b 50 08 <48> 89 51 08 48 89 0a 48 b9 22 01 00 00 00 00
 ad de 48 89 30 48 89
Oct  9 11:45:52 pbs kernel: [204607.202658] RSP: 0018:ffffb9e0c1efbcf0 EFLAGS: 00010296
Oct  9 11:45:52 pbs kernel: [204607.203504] RAX: ffff8d2ddaa9cd00 RBX: ffff8d2ddaa9ccf8 RCX: dead000000000100
Oct  9 11:45:52 pbs kernel: [204607.204520] RDX: dead000000000122 RSI: dead000000000100 RDI: ffff8d2ddaa9cd30
Oct  9 11:45:52 pbs kernel: [204607.205353] RBP: ffffb9e0c1efbd20 R08: 0000000000000001 R09: 000005665594e000
Oct  9 11:45:52 pbs kernel: [204607.206210] R10: 0000000000000003 R11: 0000000000000084 R12: ffff8d2d0d691000
Oct  9 11:45:52 pbs kernel: [204607.207065] R13: ffff8d2ddaa9cd30 R14: ffff8d2ddaa9cd10 R15: ffff8d2dc5642f80
Oct  9 11:45:52 pbs kernel: [204607.207952] FS:  0000000000000000(0000) GS:ffff8d2e3bc00000(0000) knlGS:0000000000000000
Oct  9 11:45:52 pbs kernel: [204607.208814] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  9 11:45:52 pbs kernel: [204607.209669] CR2: 00007f7007e17430 CR3: 0000000119202000 CR4: 00000000000006f0
Oct  9 11:45:52 pbs kernel: [204607.210628] Call Trace:
Oct  9 11:45:52 pbs kernel: [204607.211810]  <TASK>
Oct  9 11:45:52 pbs kernel: [204607.213082]  vdev_queue_agg_io_done+0x15/0x20 [zfs]
Oct  9 11:45:52 pbs kernel: [204607.214252]  zio_done+0x40b/0x1290 [zfs]
Oct  9 11:45:52 pbs kernel: [204607.215114]  ? zio_wait_for_children+0xb7/0x140 [zfs]
Oct  9 11:45:52 pbs kernel: [204607.215992]  zio_execute+0x95/0x160 [zfs]
Oct  9 11:45:52 pbs kernel: [204607.217319]  taskq_thread+0x29f/0x4d0 [spl]
Oct  9 11:45:52 pbs kernel: [204607.219002]  ? wake_up_q+0x90/0x90
Oct  9 11:45:52 pbs kernel: [204607.221137]  ? zio_gang_tree_free+0x70/0x70 [zfs]
Oct  9 11:45:52 pbs kernel: [204607.222145]  ? taskq_thread_spawn+0x60/0x60 [spl]
Oct  9 11:45:52 pbs kernel: [204607.223017]  kthread+0x12a/0x150
Oct  9 11:45:52 pbs kernel: [204607.224168]  ? set_kthread_struct+0x50/0x50
Oct  9 11:45:52 pbs kernel: [204607.224978]  ret_from_fork+0x22/0x30
Oct  9 11:45:52 pbs kernel: [204607.226262]  </TASK>
Oct  9 11:45:52 pbs kernel: [204607.227057] Modules linked in: target_core_user uio target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod bonding tls bochs input_leds pcspkr serio_raw drm_vram_helper efi_pstore drm_ttm_helper ttm joydev snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec drm_kms_helper snd_hda_core snd_hwdep snd_pcm cec qemu_fw_cfg mac_hid rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt snd_timer snd soundcore zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 simplefb hid_generic usbhid hid psmouse ahci i2c_i801 i2c_smbus libahci lpc_ich virtio_net net_failover failover virtio_blk virtio_scsi ehci_pci uhci_hcd ehci_hcd
Oct  9 11:45:52 pbs kernel: [204607.238895] ---[ end trace 1a9ffbce244e4684 ]---
Oct  9 11:45:53 pbs kernel: [204607.855092] RIP: 0010:abd_free+0xd1/0x1e0 [zfs]
Oct  9 11:45:53 pbs kernel: [204607.857567] Code: 6b 38 4c 89 f7 e8 df 21 35 ec 4c 89 7b 40 49 8b 44 24 50 4c 89 ef 48 be 00 01 00 00 00 00 ad de 48 01 d8 48 8b 08 48 8b 50 08 <48> 89 51 08 48 89 0a 48 b9 22 01 00 00 00 00 ad de 48 89 30 48 89
Oct  9 11:45:53 pbs kernel: [204607.860336] RSP: 0018:ffffb9e0c1efbcf0 EFLAGS: 00010296
Oct  9 11:45:53 pbs kernel: [204607.861830] RAX: ffff8d2ddaa9cd00 RBX: ffff8d2ddaa9ccf8 RCX: dead000000000100
Oct  9 11:45:53 pbs kernel: [204607.863398] RDX: dead000000000122 RSI: dead000000000100 RDI: ffff8d2ddaa9cd30
Oct  9 11:45:53 pbs kernel: [204607.864952] RBP: ffffb9e0c1efbd20 R08: 0000000000000001 R09: 000005665594e000
Oct  9 11:45:53 pbs kernel: [204607.866440] R10: 0000000000000003 R11: 0000000000000084 R12: ffff8d2d0d691000
Oct  9 11:45:53 pbs kernel: [204607.867902] R13: ffff8d2ddaa9cd30 R14: ffff8d2ddaa9cd10 R15: ffff8d2dc5642f80
Oct  9 11:45:53 pbs kernel: [204607.869451] FS:  0000000000000000(0000) GS:ffff8d2e3bc00000(0000) knlGS:0000000000000000
Oct  9 11:45:53 pbs kernel: [204607.870955] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  9 11:45:53 pbs kernel: [204607.872448] CR2: 00007f7007e17430 CR3: 0000000119202000 CR4: 00000000000006f0

Kasulikud lisamaterjalid

TODO

ZFS kasutamine operatsioonisüsteemiga Debian v. 11

Sisukord

Sissejuhatus

Paigaldamine

zfs põhine dual boot

df -h vs zfs list

2022 kevadel zfs kasutamine

ZFS failisüsteem on liiga täis ja läks katki

zpool sisu kopeerimine teisse zpool'i

Misc

zvol moodustamine ja kasutamine

zvol suuruste arvutamine

2022 sügis Promxox backup server 2.2

Kasulikud lisamaterjalid

Navigeerimismenüü

Lehetoimingud

Page actions

Isiklikud lehed

Navigeerimine

Otsing

Tööriistad