Proxmox v. 9 kasutamine

Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti

Sissejuhatus

TODO

Virtuaalne riistvara

Klaviatuur ja hiir 

root@pwrk-02:~# systool -b serio
Bus = "serio"

  Device = "serio0"
  Device = "serio1"

root@pwrk-02:~# systool -b serio -v
Bus = "serio"

  Device = "serio0"
  Device path = "/sys/devices/platform/i8042/serio0"
    bind_mode           = "auto"
    description         = "i8042 KBD port"
    drvctl              = <store method only>
    err_count           = "0"
    extra               = "0"
    firmware_id         = "PNP: PNP0303"
    force_release       = "369-370"
    modalias            = "serio:ty06pr00id00ex00"
    scroll              = "0"
    set                 = "2"
    softraw             = "1"
    softrepeat          = "0"
    uevent              = "DRIVER=atkbd
SERIO_TYPE=06
SERIO_PROTO=00
SERIO_ID=00
SERIO_EXTRA=00
MODALIAS=serio:ty06pr00id00ex00
SERIO_FIRMWARE_ID=PNP: PNP0303"

  Device = "serio1"
  Device path = "/sys/devices/platform/i8042/serio1"
    bind_mode           = "auto"
    description         = "i8042 AUX port"
    drvctl              = <store method only>
    firmware_id         = "PNP: PNP0f13"
    modalias            = "serio:ty01pr00id00ex00"
    protocol            = "VirtualPS/2"
    rate                = "100"
    resetafter          = "5"
    resolution          = "200"
    resync_time         = "0"
    uevent              = "DRIVER=psmouse
SERIO_TYPE=01
SERIO_PROTO=00
SERIO_ID=00
SERIO_EXTRA=00
MODALIAS=serio:ty01pr00id00ex00
SERIO_FIRMWARE_ID=PNP: PNP0f13"

remote zfs over iscsi

Mõisted

  • FUA (Forced Unit Access) -
  • WCE (Write Cache Enable) -
  • TPU (Thin Provisioning UNMAP) -
  • TPWS (WRITE_SAME / Zeroing) -
  • VFS (virtual file system) -

Tööpõhimõte

Virtuaalse arvuti vaatest andmetega tegelemine 

pve virtuaalne arvuti -> pve füüsiline arvuti -> over-iscsi-zfs-storage-server

Sõltuvalt qemu virtuaalse arvuti virtuaalse plokkseadme 'Cache' seadistustest on võimalik erinevate cache komplektide-kombinatsioonide kasutamine andmete liikumisel virtuaalses arvutis töötava protsessi juurest zfs storage serveri füüsilisele plokkseadmele. Tundub, et 2025 aastal on kõige keskmisemale kasutusele sobiv valik 'Cache: nocache'

  • hea jõudlus
  • hea terviklus

Andmete liikumise teekond

  • virtuaalse arvuti rakendus
  • virtuaalse arvuti vfs failisüsteem
  • virtuaalse arvuti page cache (vfs'ga seotud)
  • virtuaalse arvuti ext4 failisüsteem (ext4 draiver)
  • virtuaalse arvuti lvm
  • virtuaalse arvuti fdisk partitsioonid
  • virtuaalse arvuti virtuaalne sata vms kontroller
  • füüsilise arvuti qemu protsess
  • füüsilise arvuti open-iscsi poolt teostatud /dev/sda scsi plokkseade (puudub cache)
  • füüsilises arvutis storage'ga seotud cache puudub
  • storage arvuti targetcli (puutub cache)
  • storage arvuti zfs ressurss (sisaldab zfs cache)
  • storage avuti füüsilise plokkseadme controller-cache

'Cache: nocache' parameeter on üks vähestest parameetritest, mis avaldab mõju nö mõlemas suunas

  • virtuaalse arvuti seest kasutada olevale storage ressursile
  • kuidas füüsilise arvuti sees virtuaalsele arvutile vastav qemu protsess kasutab allolevat plokkseadet

Käesolevas tekstis kirjeldatakse nelja komponendi seadistamist

  • virtuaalsele arvutile vastav qemu protsess
  • iscsi initiator
  • iscsi target
  • zfs lahendus

zfs lahendus iscsi target arvutis

Füüsilised kettad on kasutusel nö kõige tavalisemal viisil, eriti tähendab see, et ketastel on sisselülitatud nö tavaline controller-cache 

root@pve-svc-02:~# lsscsi -s | grep 4.00T
[2:0:0:0]    disk    ATA      CT4000MX500SSD1  045   /dev/sdc   4.00TB
[3:0:0:0]    disk    ATA      CT4000MX500SSD1  045   /dev/sdd   4.00TB
[N:0:1:1]    disk    Samsung SSD 990 PRO with Heatsink 4TB__1   /dev/nvme0n1  4.00TB
[N:1:1:1]    disk    Samsung SSD 990 PRO with Heatsink 4TB__1   /dev/nvme1n1  4.00TB

zfs lülituse moodustamine 

root@pve-svc-02:~# cat create-zpool-raidz1-x4.sh
zpool create -o ashift=13 zp_data raidz1 \
  /dev/disk/by-id/ata-CT4000MX500SSD1_2246E686FE58 \
  /dev/disk/by-id/ata-CT4000MX500SSD1_2246E686FF7B \
  /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_4TB_S7DSNJ0X501827B \
  /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_4TB_S7DSNJ0X501856Z

zfs cache töötab tavalisel viisil, st on sisse lülitatud olekus, nt 

root@pve-svc-02:~# zfs get all | grep -i cache | grep vm-106-disk-0
zp_crucial_mx_4/vm-106-disk-0     primarycache          all                     default
zp_crucial_mx_4/vm-106-disk-0     secondarycache        all                     default

targetcli iscsi target arvutis

targetcli seadistamine koosneb kahest tegevusest

  • zfs põhise alus-storage storage publitseerimine
  • publitseerimisel sobivalt scsi protokolli parameetrite kasutamine (eriti cache, sparse jms esitamine)
/> /iscsi create iqn.2025-10.moraal.srv:storage.zfstarget
/> /iscsi/iqn.2025-10.moraal.srv:storage.zfstarget/tpg1/acls  create iqn.1993-08.org.debian:01:4cbe32bd26b

Sobiv seadistus, mis tuleb plokk-ressursi jaoks kehtestada 

root@pve-svc-02:~# targetcli /backstores/block/zp_crucial_mx_4-vm-108613-disk-4 get attribute | grep =
======================
alua_support=1
block_size=512
emulate_3pc=1
emulate_caw=1
emulate_dpo=1
emulate_fua_read=1
emulate_fua_write=1
emulate_model_alias=1
emulate_pr=1
emulate_rest_reord=0
emulate_rsoc=1
emulate_tas=1
emulate_tpu=0
emulate_tpws=0
emulate_ua_intlck_ctrl=0
emulate_write_cache=0
enforce_pr_isids=1
force_pr_aptpl=0
hw_block_size=512 [ro]
hw_max_sectors=32768 [ro]
hw_pi_prot_type=0 [ro]
hw_queue_depth=128 [ro]
is_nonrot=1
max_unmap_block_desc_count=1
max_unmap_lba_count=131072
max_write_same_len=65535
optimal_sectors=32768
pgr_support=1
pi_prot_format=0
pi_prot_type=0
pi_prot_verify=0
queue_depth=128
submit_type=0
unmap_granularity=8
unmap_granularity_alignment=0
unmap_zeroes_data=0

kus

  • TODO

Muudatuse tegemine 

/backstores/b...108613-disk-4> set attribute emulate_tpws=0
/backstores/b...108613-disk-4> set attribute emulate_tpu=0

Kasutamine paistab välja selline 

root@pve-svc-02:~# targetcli
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 3]
  | | o- zp_crucial_mx_4-vm-100-disk-0 ......................... [/dev/zp_crucial_mx_4/vm-100-disk-0 (20.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- zp_crucial_mx_4-vm-106-disk-5 ......................... [/dev/zp_crucial_mx_4/vm-106-disk-5 (16.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- zp_crucial_mx_4-vm-108613-disk-1 .................... [/dev/zp_crucial_mx_4/vm-108613-disk-1 (4.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2025-10.moraal.srv:storage.zfstarget .......................................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.1993-08.org.debian:01:4cbe32bd26b ................................................................ [Mapped LUNs: 3]
  |     |   o- mapped_lun0 ...................................................... [lun0 block/zp_crucial_mx_4-vm-108613-disk-1 (rw)]
  |     |   o- mapped_lun1 ......................................................... [lun1 block/zp_crucial_mx_4-vm-106-disk-5 (rw)]
  |     |   o- mapped_lun2 ......................................................... [lun2 block/zp_crucial_mx_4-vm-100-disk-0 (rw)]
  |     o- luns .......................................................................................................... [LUNs: 3]
  |     | o- lun0 .............. [block/zp_crucial_mx_4-vm-108613-disk-1 (/dev/zp_crucial_mx_4/vm-108613-disk-1) (default_tg_pt_gp)]
  |     | o- lun1 .................... [block/zp_crucial_mx_4-vm-106-disk-5 (/dev/zp_crucial_mx_4/vm-106-disk-5) (default_tg_pt_gp)]
  |     | o- lun2 .................... [block/zp_crucial_mx_4-vm-100-disk-0 (/dev/zp_crucial_mx_4/vm-100-disk-0) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- srpt ............................................................................................................. [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]
  o- xen-pvscsi ....................................................................................................... [Targets: 0]
/>

Lisaks saab portaali käest küsida attribute ja parameter komplekte, nt 

root@pm60-trt:~# targetcli /iscsi/iqn.2022-09.ee.moraal:pbs-pub/tpg1 get parameter | grep =
======================
AuthMethod=CHAP
DataDigest=CRC32C,None
DataPDUInOrder=Yes
DataSequenceInOrder=Yes
DefaultTime2Retain=20
DefaultTime2Wait=2
ErrorRecoveryLevel=0
FirstBurstLength=65536
HeaderDigest=CRC32C,None
IFMarkInt=Reject
IFMarker=No
ImmediateData=Yes
InitialR2T=Yes
MaxBurstLength=262144
MaxConnections=1
MaxOutstandingR2T=1
MaxRecvDataSegmentLength=8192
MaxXmitDataSegmentLength=262144
OFMarkInt=Reject
OFMarker=No
TargetAlias=LIO Target

root@pm60-trt:~# targetcli /iscsi/iqn.2022-09.ee.moraal:pbs-pub/tpg1 get attribute | grep =
======================
authentication=1
cache_dynamic_acls=0
default_cmdsn_depth=64
default_erl=0
demo_mode_discovery=1
demo_mode_write_protect=1
fabric_prot_type=0
generate_node_acls=0
login_keys_workaround=1
login_timeout=15
prod_mode_write_protect=0
t10_pi=0
tpg_enabled_sendtargets=1
root@pm60-trt:~#

kus

  • parameter - võrgus kõneldavad asjad
  • attribute - lokaalselt olukorda määratlevad asjad

iscsi initiator

TODO

qemu protsess

Kasutamine

20251019-remote-zfs-over-iscsi-01.png

Kasulikud lisamaterjalid

TODO

Secure boot - nn Microsoft 2023 sertifikaat

Tööpõhimõte

TODO

Olemasolev olukord enne muudatust

root@pwrk-01:~# apt-get install efitools
root@pwrk-01:~# (printf "db: \n"; efi-readvar -v db; printf "\nKEK: \n"; efi-readvar -v KEK) | grep -E "2011|2023"
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation UEFI CA 2011
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation KEK CA 2011

Muudatus

root@pve-wrx90e:~# qm enroll-efi-keys 902198
skipping - OS type is neither Windows 10 nor Windows 11

root@pve-wrx90e:~# qm set 902198 --ostype win10
root@pve-wrx90e:~# qm enroll-efi-keys 902198
root@pve-wrx90e:~# qm set 902198 --ostype l26

ja 

from

efidisk0: si-dpool:vm-902198-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M

to

efidisk0: si-dpool:vm-902198-disk-0,efitype=4m,ms-cert=2023w,pre-enrolled-keys=1,size=1M

Uus olukord peale muudatust

root@pwrk-02:~# (printf "db: \n"; efi-readvar -v db; printf "\nKEK: \n"; efi-readvar -v KEK) | grep -E "2011|2023"
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation UEFI CA 2011
            C=US, O=Microsoft Corporation, CN=Microsoft UEFI CA 2023
            C=US, O=Microsoft Corporation, CN=Windows UEFI CA 2023
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation KEK CA 2011

Misc

pveupdate

  • uuendab webgui liidese sertifikaadi
root@pve-wrx90e:~# pveupdate
Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/2232348225/504476182801

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz/2232348225/694056506271'
The validation for pve-wrx90e.auul.pri.ee is pending!
Setting up webserver
Triggering validation
Sleeping for 5 seconds
Status is 'valid', domain 'pve-wrx90e.auul.pri.ee' OK!

All domains validated!

Creating CSR
Checking order status
Order is ready, finalizing order
valid!

Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
Revoking old certificate
Revoke request to CA failed: Error: POST to https://acme-v02.api.letsencrypt.org/acme/revoke-cert
{
  "type": "urn:ietf:params:acme:error:unauthorized",
  "detail": "Unable to revoke :: Certificate is expired",
  "status": 403
}

pveupgrade

  • uuendab süsteemi apt paketihaldusega tarkvara

Kasulikud lisamaterjalid

  • TODO
Pärit leheküljelt "https://www.auul.pri.ee/wiki/index.php?title=Proxmox_v._9_kasutamine&oldid=3317"