Proxmox v. 9 kasutamine

Allikas: Imre kasutab arvutit
Redaktsioon seisuga 17. mai 2026, kell 14:22 kasutajalt Imre (arutelu | kaastöö) (→‎Muudatuse protseduur)
Mine navigeerimisribaleMine otsikasti

Sissejuhatus

TODO

Virtuaalne riistvara

Klaviatuur ja hiir 

root@pwrk-02:~# systool -b serio
Bus = "serio"

  Device = "serio0"
  Device = "serio1"

root@pwrk-02:~# systool -b serio -v
Bus = "serio"

  Device = "serio0"
  Device path = "/sys/devices/platform/i8042/serio0"
    bind_mode           = "auto"
    description         = "i8042 KBD port"
    drvctl              = <store method only>
    err_count           = "0"
    extra               = "0"
    firmware_id         = "PNP: PNP0303"
    force_release       = "369-370"
    modalias            = "serio:ty06pr00id00ex00"
    scroll              = "0"
    set                 = "2"
    softraw             = "1"
    softrepeat          = "0"
    uevent              = "DRIVER=atkbd
SERIO_TYPE=06
SERIO_PROTO=00
SERIO_ID=00
SERIO_EXTRA=00
MODALIAS=serio:ty06pr00id00ex00
SERIO_FIRMWARE_ID=PNP: PNP0303"

  Device = "serio1"
  Device path = "/sys/devices/platform/i8042/serio1"
    bind_mode           = "auto"
    description         = "i8042 AUX port"
    drvctl              = <store method only>
    firmware_id         = "PNP: PNP0f13"
    modalias            = "serio:ty01pr00id00ex00"
    protocol            = "VirtualPS/2"
    rate                = "100"
    resetafter          = "5"
    resolution          = "200"
    resync_time         = "0"
    uevent              = "DRIVER=psmouse
SERIO_TYPE=01
SERIO_PROTO=00
SERIO_ID=00
SERIO_EXTRA=00
MODALIAS=serio:ty01pr00id00ex00
SERIO_FIRMWARE_ID=PNP: PNP0f13"

remote zfs over iscsi

Mõisted

  • FUA (Forced Unit Access) -
  • WCE (Write Cache Enable) -
  • TPU (Thin Provisioning UNMAP) -
  • TPWS (WRITE_SAME / Zeroing) -
  • VFS (virtual file system) -

Tööpõhimõte

Virtuaalse arvuti vaatest andmetega tegelemine 

pve virtuaalne arvuti -> pve füüsiline arvuti -> over-iscsi-zfs-storage-server

Sõltuvalt qemu virtuaalse arvuti virtuaalse plokkseadme 'Cache' seadistustest on võimalik erinevate cache komplektide-kombinatsioonide kasutamine andmete liikumisel virtuaalses arvutis töötava protsessi juurest zfs storage serveri füüsilisele plokkseadmele. Tundub, et 2025 aastal on kõige keskmisemale kasutusele sobiv valik 'Cache: nocache'

  • hea jõudlus
  • hea terviklus

Andmete liikumise teekond

  • virtuaalse arvuti rakendus
  • virtuaalse arvuti vfs failisüsteem
  • virtuaalse arvuti page cache (vfs'ga seotud)
  • virtuaalse arvuti ext4 failisüsteem (ext4 draiver)
  • virtuaalse arvuti lvm
  • virtuaalse arvuti fdisk partitsioonid
  • virtuaalse arvuti virtuaalne sata vms kontroller
  • füüsilise arvuti qemu protsess
  • füüsilise arvuti open-iscsi poolt teostatud /dev/sda scsi plokkseade (puudub cache)
  • füüsilises arvutis storage'ga seotud cache puudub
  • storage arvuti targetcli (puutub cache)
  • storage arvuti zfs ressurss (sisaldab zfs cache)
  • storage avuti füüsilise plokkseadme controller-cache

'Cache: nocache' parameeter on üks vähestest parameetritest, mis avaldab mõju nö mõlemas suunas

  • virtuaalse arvuti seest kasutada olevale storage ressursile
  • kuidas füüsilise arvuti sees virtuaalsele arvutile vastav qemu protsess kasutab allolevat plokkseadet

Käesolevas tekstis kirjeldatakse nelja komponendi seadistamist

  • virtuaalsele arvutile vastav qemu protsess
  • iscsi initiator
  • iscsi target
  • zfs lahendus

zfs lahendus iscsi target arvutis

Füüsilised kettad on kasutusel nö kõige tavalisemal viisil, eriti tähendab see, et ketastel on sisselülitatud nö tavaline controller-cache 

root@pve-svc-02:~# lsscsi -s | grep 4.00T
[2:0:0:0]    disk    ATA      CT4000MX500SSD1  045   /dev/sdc   4.00TB
[3:0:0:0]    disk    ATA      CT4000MX500SSD1  045   /dev/sdd   4.00TB
[N:0:1:1]    disk    Samsung SSD 990 PRO with Heatsink 4TB__1   /dev/nvme0n1  4.00TB
[N:1:1:1]    disk    Samsung SSD 990 PRO with Heatsink 4TB__1   /dev/nvme1n1  4.00TB

zfs lülituse moodustamine 

root@pve-svc-02:~# cat create-zpool-raidz1-x4.sh
zpool create -o ashift=13 zp_data raidz1 \
  /dev/disk/by-id/ata-CT4000MX500SSD1_2246E686FE58 \
  /dev/disk/by-id/ata-CT4000MX500SSD1_2246E686FF7B \
  /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_4TB_S7DSNJ0X501827B \
  /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_with_Heatsink_4TB_S7DSNJ0X501856Z

zfs cache töötab tavalisel viisil, st on sisse lülitatud olekus, nt 

root@pve-svc-02:~# zfs get all | grep -i cache | grep vm-106-disk-0
zp_crucial_mx_4/vm-106-disk-0     primarycache          all                     default
zp_crucial_mx_4/vm-106-disk-0     secondarycache        all                     default

targetcli iscsi target arvutis

targetcli seadistamine koosneb kahest tegevusest

  • zfs põhise alus-storage storage publitseerimine
  • publitseerimisel sobivalt scsi protokolli parameetrite kasutamine (eriti cache, sparse jms esitamine)
/> /iscsi create iqn.2025-10.moraal.srv:storage.zfstarget
/> /iscsi/iqn.2025-10.moraal.srv:storage.zfstarget/tpg1/acls  create iqn.1993-08.org.debian:01:4cbe32bd26b

Sobiv seadistus, mis tuleb plokk-ressursi jaoks kehtestada 

root@pve-svc-02:~# targetcli /backstores/block/zp_crucial_mx_4-vm-108613-disk-4 get attribute | grep =
======================
alua_support=1
block_size=512
emulate_3pc=1
emulate_caw=1
emulate_dpo=1
emulate_fua_read=1
emulate_fua_write=1
emulate_model_alias=1
emulate_pr=1
emulate_rest_reord=0
emulate_rsoc=1
emulate_tas=1
emulate_tpu=0
emulate_tpws=0
emulate_ua_intlck_ctrl=0
emulate_write_cache=0
enforce_pr_isids=1
force_pr_aptpl=0
hw_block_size=512 [ro]
hw_max_sectors=32768 [ro]
hw_pi_prot_type=0 [ro]
hw_queue_depth=128 [ro]
is_nonrot=1
max_unmap_block_desc_count=1
max_unmap_lba_count=131072
max_write_same_len=65535
optimal_sectors=32768
pgr_support=1
pi_prot_format=0
pi_prot_type=0
pi_prot_verify=0
queue_depth=128
submit_type=0
unmap_granularity=8
unmap_granularity_alignment=0
unmap_zeroes_data=0

kus

  • TODO

Muudatuse tegemine 

/backstores/b...108613-disk-4> set attribute emulate_tpws=0
/backstores/b...108613-disk-4> set attribute emulate_tpu=0

Kasutamine paistab välja selline 

root@pve-svc-02:~# targetcli
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 3]
  | | o- zp_crucial_mx_4-vm-100-disk-0 ......................... [/dev/zp_crucial_mx_4/vm-100-disk-0 (20.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- zp_crucial_mx_4-vm-106-disk-5 ......................... [/dev/zp_crucial_mx_4/vm-106-disk-5 (16.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- zp_crucial_mx_4-vm-108613-disk-1 .................... [/dev/zp_crucial_mx_4/vm-108613-disk-1 (4.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2025-10.moraal.srv:storage.zfstarget .......................................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.1993-08.org.debian:01:4cbe32bd26b ................................................................ [Mapped LUNs: 3]
  |     |   o- mapped_lun0 ...................................................... [lun0 block/zp_crucial_mx_4-vm-108613-disk-1 (rw)]
  |     |   o- mapped_lun1 ......................................................... [lun1 block/zp_crucial_mx_4-vm-106-disk-5 (rw)]
  |     |   o- mapped_lun2 ......................................................... [lun2 block/zp_crucial_mx_4-vm-100-disk-0 (rw)]
  |     o- luns .......................................................................................................... [LUNs: 3]
  |     | o- lun0 .............. [block/zp_crucial_mx_4-vm-108613-disk-1 (/dev/zp_crucial_mx_4/vm-108613-disk-1) (default_tg_pt_gp)]
  |     | o- lun1 .................... [block/zp_crucial_mx_4-vm-106-disk-5 (/dev/zp_crucial_mx_4/vm-106-disk-5) (default_tg_pt_gp)]
  |     | o- lun2 .................... [block/zp_crucial_mx_4-vm-100-disk-0 (/dev/zp_crucial_mx_4/vm-100-disk-0) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- srpt ............................................................................................................. [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]
  o- xen-pvscsi ....................................................................................................... [Targets: 0]
/>

Lisaks saab portaali käest küsida attribute ja parameter komplekte, nt 

root@pm60-trt:~# targetcli /iscsi/iqn.2022-09.ee.moraal:pbs-pub/tpg1 get parameter | grep =
======================
AuthMethod=CHAP
DataDigest=CRC32C,None
DataPDUInOrder=Yes
DataSequenceInOrder=Yes
DefaultTime2Retain=20
DefaultTime2Wait=2
ErrorRecoveryLevel=0
FirstBurstLength=65536
HeaderDigest=CRC32C,None
IFMarkInt=Reject
IFMarker=No
ImmediateData=Yes
InitialR2T=Yes
MaxBurstLength=262144
MaxConnections=1
MaxOutstandingR2T=1
MaxRecvDataSegmentLength=8192
MaxXmitDataSegmentLength=262144
OFMarkInt=Reject
OFMarker=No
TargetAlias=LIO Target

root@pm60-trt:~# targetcli /iscsi/iqn.2022-09.ee.moraal:pbs-pub/tpg1 get attribute | grep =
======================
authentication=1
cache_dynamic_acls=0
default_cmdsn_depth=64
default_erl=0
demo_mode_discovery=1
demo_mode_write_protect=1
fabric_prot_type=0
generate_node_acls=0
login_keys_workaround=1
login_timeout=15
prod_mode_write_protect=0
t10_pi=0
tpg_enabled_sendtargets=1
root@pm60-trt:~#

kus

  • parameter - võrgus kõneldavad asjad
  • attribute - lokaalselt olukorda määratlevad asjad

iscsi initiator

TODO

qemu protsess

Kasutamine

20251019-remote-zfs-over-iscsi-01.png

Kasulikud lisamaterjalid

TODO

Secure boot - nn Microsoft 2023 sertifikaat

Tööpõhimõte

TODO

Olemasolev olukord enne muudatust

root@pwrk-01:~# apt-get install efitools
root@pwrk-01:~# (printf "db: \n"; efi-readvar -v db; printf "\nKEK: \n"; efi-readvar -v KEK) | grep -E "2011|2023"
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation UEFI CA 2011
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation KEK CA 2011

Muudatus

root@pve-wrx90e:~# qm enroll-efi-keys 902198
skipping - OS type is neither Windows 10 nor Windows 11

root@pve-wrx90e:~# qm set 902198 --ostype win10
root@pve-wrx90e:~# qm enroll-efi-keys 902198
root@pve-wrx90e:~# qm set 902198 --ostype l26

ja 

from

efidisk0: si-dpool:vm-902198-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M

to

efidisk0: si-dpool:vm-902198-disk-0,efitype=4m,ms-cert=2023w,pre-enrolled-keys=1,size=1M

Uus olukord peale muudatust

root@pwrk-02:~# (printf "db: \n"; efi-readvar -v db; printf "\nKEK: \n"; efi-readvar -v KEK) | grep -E "2011|2023"
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation UEFI CA 2011
            C=US, O=Microsoft Corporation, CN=Microsoft UEFI CA 2023
            C=US, O=Microsoft Corporation, CN=Windows UEFI CA 2023
            C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Corporation KEK CA 2011

Misc

pveupdate

  • uuendab webgui liidese sertifikaadi
root@pve-wrx90e:~# pveupdate
Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/2232348225/504476182801

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz/2232348225/694056506271'
The validation for pve-wrx90e.auul.pri.ee is pending!
Setting up webserver
Triggering validation
Sleeping for 5 seconds
Status is 'valid', domain 'pve-wrx90e.auul.pri.ee' OK!

All domains validated!

Creating CSR
Checking order status
Order is ready, finalizing order
valid!

Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
Revoking old certificate
Revoke request to CA failed: Error: POST to https://acme-v02.api.letsencrypt.org/acme/revoke-cert
{
  "type": "urn:ietf:params:acme:error:unauthorized",
  "detail": "Unable to revoke :: Certificate is expired",
  "status": 403
}

pveupgrade

  • uuendab süsteemi apt paketihaldusega tarkvara

BIOS arvuti teisendamine UEFI arvutiks

Tööpõhimõte

Peamised kaalutlused BIOS arvuti teisendamisel UEFI arvutiks

  • vajadus saada osa secure boot lahendusega kaasnevast turvalisusest
  • nö täieline reinstall oleks liiga aeganõudev, kohmakas jne

Muudatus seisneb virtuaalsele arvutile täiendava nö esimese plokkseadme lisamisel, suurusega 1 G. Sinna moodustatakse gpt partitsioonitabel ning vfat failisüsteem, ja ta vastab nö kataloogile /boot/efi, sinna ta ka monteeritakse kokkuvõttes. Juurfailisüsteem ja muu jääb sinna kus ta seni on olnud. Oluline on, et muudatuse käigus et muudeta olemasoleva plokkseadme lvm ega partitsioonitabelit, st ebaõnnestumisel saab pöörduda tagasi kergesti (varundus peaks aga siiski olema olemas).

Muudatuse protseduur

Enne muudatust paistab virtuaalne arvuti selline

20260516-from-bios-to-uefi-01.png

ning 

# df -T -h -t ext4
Failisüsteem                      Tüüp  Maht Kasut  Vaba Kas% Haagitud
/dev/mapper/system-root           ext4   11G  8,2G  2,2G  80% /
/dev/vda1                         ext4  462M  325M  109M  75% /boot

Veenduda, mis on konkreetsel juhtumil 1 g lisatud plokkseadme nimi, nt 

# fdisk /dev/vdb -l
Disk /dev/vdb: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

edasi käsundamine 

# apt-get install parted

# parted /dev/vdb mklabel gpt
# parted /dev/vdb mkpart primary fat32 1MiB 100%
# parted /dev/vdb set 1 esp on
# mkfs.vfat -F 32 /dev/vdb1
# mkdir -p /boot/efi
# blkid /dev/vdb1

Muuta /etc/fstab faili, st lisada üks rida olemaoleva /boot rea alla, uuid on mitte partitsiooni, aga vfat failisüsteemi 

# cat /etc/fstab
..
UUID=XXXX-XXXX  /boot/efi       vfat    defaults        0       2

Lisada paketid 

# systemctl daemon-reload
# mount /boot/efi
# apt update
# apt install grub-efi-amd64-signed shim-signed mokutil
# grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian --removable
# update-grub

tulemusena 

# find /boot/efi/ -type f -ls
      115    932 -rwxr-xr-x   1 root     root       952384 mai   17 12:00 /boot/efi/EFI/BOOT/BOOTX64.EFI
      116   2624 -rwxr-xr-x   1 root     root      2685544 mai   17 12:00 /boot/efi/EFI/BOOT/grubx64.efi
      117    832 -rwxr-xr-x   1 root     root       851368 mai   17 12:00 /boot/efi/EFI/BOOT/mmx64.efi
      118      4 -rwxr-xr-x   1 root     root          112 mai   17 12:00 /boot/efi/EFI/BOOT/BOOTX64.CSV
      119      4 -rwxr-xr-x   1 root     root          112 mai   17 12:00 /boot/efi/EFI/BOOT/grub.cfg

# cat /boot/efi/EFI/BOOT/grub.cfg
search.fs_uuid 5ebbc6f0-69e8-413b-bb4f-4ec0fa5d2fc1 root
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

# blkid /dev/vda1
/dev/vda1: UUID="5ebbc6f0-69e8-413b-bb4f-4ec0fa5d2fc1" BLOCK_SIZE="1024" TYPE="ext4" PARTUUID="cfa36d3a-01"

PVE virtuaalsele arvutile

  • lisada 'EFI Disk' (virtuaalne efi riistvara seadistuste salvestamine jms)
  • bios -> uefi
  • chipset 440 -> q35
  • display default -> virtio-gpu

Peale muudatust paistab virtuaalne arvuti selline

20260516-from-bios-to-uefi-02.png

Lisaks on boot järjekorra seadmed muudetud

20260516-from-bios-to-uefi-03.png

kus

  • tingimata vajalik on virtio1 ja virtio0 - esimesel on bootload ja grub.cfg lühike konf, teisel on konfi sisuline osa

Tulemusena töötab arvuti uefi secure boot režiimis 

# mokutil --sb-state
SecureBoot enabled

Kõige lõpus võiks veel öelda, nii tekivad nö naturaalsed efibootmgr ja /boot/efi sissekanded nagu nad oleks seal nö sündinud-uefi'na arvutil 

# efibootmgr
BootCurrent: 0002
Timeout: 3 seconds
BootOrder: 0002,0003,0000,0001
Boot0000* BootManagerMenuApp	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(eec25bdc-67f2-4d95-b1d5-f81b2039d11d)
Boot0001* EFI Firmware Setup	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0002* UEFI Misc Device	PciRoot(0x0)/Pci(0x1e,0x0)/Pci(0x1,0x0)/Pci(0xb,0x0){auto_created_boot_option}
Boot0003* UEFI Misc Device 2	PciRoot(0x0)/Pci(0x1e,0x0)/Pci(0x1,0x0)/Pci(0xa,0x0){auto_created_boot_option}

# grub-install /dev/vdb
Installing for x86_64-efi platform.
Installation finished. No error reported.

# efibootmgr
BootCurrent: 0002
Timeout: 3 seconds
BootOrder: 0004,0002,0003,0000,0001
Boot0000* BootManagerMenuApp	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(eec25bdc-67f2-4d95-b1d5-f81b2039d11d)
Boot0001* EFI Firmware Setup	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0002* UEFI Misc Device	PciRoot(0x0)/Pci(0x1e,0x0)/Pci(0x1,0x0)/Pci(0xb,0x0){auto_created_boot_option}
Boot0003* UEFI Misc Device 2	PciRoot(0x0)/Pci(0x1e,0x0)/Pci(0x1,0x0)/Pci(0xa,0x0){auto_created_boot_option}
Boot0004* debian	HD(1,GPT,0fb8cfd1-c90b-4da2-97db-ceda14464a90,0x800,0x1ff000)/File(\EFI\proxmox\shimx64.efi)

Kasulikud lisamaterjalid

  • TODO
Pärit leheküljelt "https://www.auul.pri.ee/wiki/index.php?title=Proxmox_v._9_kasutamine&oldid=3626"