Linux plokkseadme kasutamine

Sissejuhatus

TODO

Tööpõhimõte

TODO

Mõõtmine - üksik sata-ssd ketas

Üksiku sata-ssd kettana on kasutuses selline seade

root@rae-cf-04:~# smartctl -a /dev/sde
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-130-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     Micron_5400_MTFDDAK7T6TGA
Serial Number:    24504CD75CBC
LU WWN Device Id: 5 00a075 14cd75cbc
Firmware Version: D4MU802
User Capacity:    7 681 501 126 656 bytes [7,68 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Feb  2 00:11:16 2025 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
...

fio seadistusfail

root@rae-cf-04:~# cat fio.fio 
[global]
ioengine=libaio
invalidate=1
ramp_time=5
runtime=180
time_based
direct=1
group_reporting=1
per_job_logs=0
bs=8k
readwrite=randrw
size=40g
iodepth=32
end_fsync=1
# gtod_reduce=1
  
[test1-subtest1-blocksize32k]
filename=/mnt/sde/fio.img
numjobs=12

kus

• bs=8k - väljendab, et fio protsess teeb linux kerneli suunal sellise iseloomuga (mahuga, liigendusega, kiirusega jne) suuruste tükkidega tegevusi (ja nagu allpool on iostat väljendist paista, millele reageerib tuum plokkseadme poole 8k suurustege päringutega pöördumistega)
• TODO

ning käivitamisel paistab

root@rae-cf-04:~# iostat -m -x 30 /dev/sde | grep ^sde
Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
..
sde           19900,50    155,47     0,07   0,00    1,51     8,00 19901,57    155,48     0,33   0,00    1,55     8,00    0,00      0,00     0,00   0,00    0,00     0,00    0,40    1,67   60,91 100,00
sde           19831,50    154,93     0,03   0,00    1,52     8,00 19854,03    155,11     0,30   0,00    1,55     8,00    0,00      0,00     0,00   0,00    0,00     0,00    0,40    1,83   60,91 100,00
sde           19761,47    154,39     0,10   0,00    1,52     8,00 19830,37    154,93     0,50   0,00    1,56     8,00    0,00      0,00     0,00   0,00    0,00     0,00    0,40    2,50   60,91 100,00
sde           19995,50    156,22     0,03   0,00    1,51     8,00 19992,33    156,19     0,50   0,00    1,54     8,00    0,00      0,00     0,00   0,00    0,00     0,00    0,40    2,08   60,91 100,00
sde           19697,80    153,89     0,10   0,00    1,52     8,00 19723,87    154,09     0,33   0,00    1,57     8,00    0,00      0,00     0,00   0,00    0,00     0,00    0,40    1,92   60,91 100,00
sde           19959,23    155,93     0,07   0,00    1,51     8,00 19939,47    155,78     0,43   0,00    1,54     8,00    0,00      0,00     0,00   0,00    0,00     0,00    0,40    1,67   60,91 100,00
sde             553,23      4,32     0,00   0,00    1,54     8,00   557,47      4,35     0,07   0,01    1,67     8,00    0,00      0,00     0,00   0,00    0,00     0,00    0,13    0,75    1,78   2,95
sde               0,17      0,00     0,00   0,00    1,00     0,30     0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00    0,00    0,00   0,03
...

kus

• arvulised väärtused väljendavad linuxi kernelist füüsilise plokkseadme suunal toimuvat, nt wareq-sz (write request size) väljendab, et toimub 8 kBait suuruse plokiga kirjutamine
• wareq-sz ja rareq-sz - kasutatava ploki suurus
• w/s ja r/s - operatsioonide arv sekundis (tuntud kui iops)
• wrqm/s ja rrqm/s - write ja read request merges sekundis - antud juhul seda ei toimu praktiliselt, aga teatud tingimuste puhul linux kernel käigult agregeerib kokku sisse tulevaid kirjutamisi suuremateks komplektideks, ja saadab need korraga füüsilisele plokkseadmele
• w_await ja r_await - kirjutamise ja lugemisega seotud viivutus - üldiselt loetakse normaalseks alla 4 ms
• wMB/s ja rMB/s - mahu kirjutamise ja lugemise kiirus
• %util - kui palju ajast tegeleb ketas tööga (selle asemel et ootada korraldust töötada)
• dXXX - discard tegevused - reeglina on nad asjakohased mingit tüüpi ssd ketastega (nvme, sata-ssd, sas-ssd jms)

ning fio tulemus

root@rae-cf-04:~# fio fio.fio 
test1-subtest1-blocksize32k: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32
...
fio-3.28
Starting 12 processes
Jobs: 12 (f=12): [F(1),m(1),F(3),m(1),F(1),m(5)][11.6%][r=149MiB/s,w=149MiB/s][r=19.0k,w=19.1k IOPS][eta 23m:36s]
test1-subtest1-blocksize32k: (groupid=0, jobs=12): err= 0: pid=3266168: Sat Feb  1 23:15:16 2025
  read: IOPS=19.9k, BW=155MiB/s (163MB/s)(27.3GiB/180007msec)
    slat (usec): min=2, max=19609, avg=298.37, stdev=637.42
    clat (usec): min=363, max=51174, avg=9379.33, stdev=3656.91
     lat (usec): min=542, max=52640, avg=9677.78, stdev=3755.98
    clat percentiles (usec):
     |  1.00th=[ 3556],  5.00th=[ 4752], 10.00th=[ 5473], 20.00th=[ 6456],
     | 30.00th=[ 7242], 40.00th=[ 8029], 50.00th=[ 8717], 60.00th=[ 9503],
     | 70.00th=[10552], 80.00th=[11863], 90.00th=[13960], 95.00th=[16188],
     | 99.00th=[21627], 99.50th=[23987], 99.90th=[29230], 99.95th=[31327],
     | 99.99th=[35914]
   bw (  KiB/s): min=126053, max=189343, per=100.00%, avg=158884.11, stdev=918.85, samples=4309
   iops        : min=15756, max=23667, avg=19860.25, stdev=114.85, samples=4309
  write: IOPS=19.9k, BW=155MiB/s (163MB/s)(27.3GiB/180007msec); 0 zone resets
    slat (usec): min=2, max=18651, avg=302.93, stdev=641.72
    clat (usec): min=380, max=52645, avg=9353.20, stdev=3745.15
     lat (usec): min=384, max=53573, avg=9656.22, stdev=3846.76
    clat percentiles (usec):
     |  1.00th=[ 3261],  5.00th=[ 4555], 10.00th=[ 5342], 20.00th=[ 6390],
     | 30.00th=[ 7177], 40.00th=[ 7963], 50.00th=[ 8717], 60.00th=[ 9503],
     | 70.00th=[10552], 80.00th=[11863], 90.00th=[14091], 95.00th=[16319],
     | 99.00th=[21890], 99.50th=[23987], 99.90th=[29230], 99.95th=[31327],
     | 99.99th=[36439]
   bw (  KiB/s): min=126768, max=191136, per=100.00%, avg=159008.05, stdev=903.82, samples=4309
   iops        : min=15846, max=23892, avg=19875.76, stdev=112.97, samples=4309
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.06%, 4=2.24%, 10=62.52%, 20=33.52%, 50=1.67%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.59%, sys=1.62%, ctx=2589547, majf=0, minf=699
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=3573369,3576278,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=27.3GiB (29.3GB), run=180007-180007msec
  WRITE: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=27.3GiB (29.3GB), run=180007-180007msec

Disk stats (read/write):
  sde: ios=3674207/3677588, merge=13/73, ticks=5564118/5703731, in_queue=11267990, util=99.97%

kus

• slat - start latency
• clat - completion latency
• iops 19.9 k
• TODO

Väited

• andmesalvestuse käitumine on dünaamiline - nt suurendades fio testis fio protsessi 8k väärtust nt 8M, tulemusena on iostat väljundis wareq-sz väärtus 1024 ümbruses

Kasulikud lisamaterjalid

• 'Chapter 32. Factors affecting I/O and file system performance' - https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/factors-affecting-i-o-and-file-system-performance_monitoring-and-managing-system-status-and-performance#tools-for-monitoring-and-diagnosing-i-o-and-file-system-issues_factors-affecting-i-o-and-file-system-performance
• TODO