Terraform: erinevus redaktsioonide vahel

Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti
 
(ei näidata sama kasutaja 17 vahepealset redaktsiooni)
5. rida: 5. rida:
 
===Mõisted===
  
===Mõisted===
   
−
* ias - infrastructure as code
 +
* IaC - infrastructure as code
   
 
===Tööpõhimõte===
  
===Tööpõhimõte===
83. rida: 83. rida:
 
====2022====
  
====2022====
   
  +
Üks juhend asub aadressil https://austinsnerdythings.com/2021/09/01/how-to-deploy-vms-in-proxmox-with-terraform/ ja originaal https://www.terraform.io/cli/install/apt
−
TODO
  
  +
  +
Kopeerida tootja juurest repo pgp võti
  +
  +
<pre>
  +
tookohaarvuti# wget https://apt.releases.hashicorp.com/gpg -O /etc/apt/keyrings/hashicorp-archive-keyring.pub
  +
tookohaarvuti# cat /etc/apt/keyrings/hashicorp-archive-keyring.pub
  +
-----BEGIN PGP PUBLIC KEY BLOCK-----
  +
mQINBF60TuYBEADLS1MP7XrMlRkn1Y54cb2UclUMH8HkIRfBrhk5Leo9kNZc/2QD
  +
LmdQbi3UbZkz0uVkHqbFDgV5lAnukCnxgr9BqnL0GJpO78le7gCCbM5bR4rTJ6Ar
  +
OOtIKf25smGTIpbSwNdj8BOLqiExGFj/9L5X9S5kfq3vtuYt+lmxKkIrEPjSYnFR
  +
TQ2mTL8RM932GJod/5VJ2+6YvrCjtPu5/rW02H1U2ZHiTtX6ZGnIvv/sprKyFRqT
  +
x4Ib+o9XwXof/LuxTMpVwIHSzCYanH5hPc7yRGKzIntBS+dDom+h9smx7FTgpHwt
  +
QRFGLtVoHXqON6nXTLFDkEzxr+fXq/bgB1Kc1TuzvoK601ztQGhhDaEPloKqNWM8
  +
Ho7JU1RpnoWr5jOFTYiPM9uyCtFNsJmD9mt4K8sQQN7T2inR5Us0o510FqePRFeX
  +
wOJUMi1CbeYqVHfKQ5cWYujcK8pv3l1a6dSBmFfcdxtwIoA16JzCrgsCeumTDvKu
  +
hOiTctb28srL/9WwlijUzZy6R2BGBbhP937f2NbMS/rpby7M1WizKeo2tkKVyK+w
  +
SUWSw6EtFJi7kRSkH7rvy/ysU9I2ma88TyvyOgIz1NRRXYsW7+brgwXnuJraOLaB
  +
5aiuhlngKpTPvP9CFib7AW2QOXustMZ7pOUREmxgS4kqxo74CuFws163TwARAQAB
  +
tFFIYXNoaUNvcnAgU2VjdXJpdHkgKEhhc2hpQ29ycCBQYWNrYWdlIFNpZ25pbmcp
  +
IDxzZWN1cml0eStwYWNrYWdpbmdAaGFzaGljb3JwLmNvbT6JAk4EEwEIADgWIQTo
  +
oDLglNjrTqGJ0nDaQYyIoyGfewUCXrRO5gIbAwULCQgHAgYVCgkICwIEFgIDAQIe
  +
AQIXgAAKCRDaQYyIoyGfe6/WD/9dTM/1OSgbvSPpPJOOcn5L1nOKRBJpztr4V0ky
  +
GoCDakIQ/sykbcuHXP79FGLzrM8zQOsbvVp/Z2lsWBnxkT8KWM+8LZxYToRGdZhr
  +
huFPHV9df0vAsZGisu4ejHDneHOTO3KqVotkky34jUSjBL7Q8uwXHY9r+5hb452N
  +
vafN1w0Y1QVhb6JjjwWHR8Rf9qkSIEi6m9o8a1M54yQC2y/Zrs6+4F3zZ4uYfTvz
  +
MyFfj0P5VmAoaowLSRdb2/JTObu0+zpKN+PjZA8BcnOf/pvqmEz83FIfo6zJLScx
  +
TVaAwj5Iz/jS04x7EvBuIP3vpgv1R6r+t0qU/7hpu7Oc0dsxhL+C8BpVY26/2hvX
  +
ozN5eG0ysSwexqwls+bnRgd6KdoHlWFNfbW8RCPKyb/s+tmFqGAY/QmxMkukgnXQ
  +
WvBoa0Gdv2AFVLYup9tEO1zF4zBPh5oQwAXDNudLTHJ4KmyEwWsOQJUjNB4y4a7j
  +
iGgK77T4KKXpo7pVDP8Ur+tmNH/d+/YFjxrfJvWt4ypE5dZmFO/FrUMvIGglOLDt
  +
A+SiQe73IpEebB8PiqNlqJ2NU7artuRxYQVColt+/1puIHwV+h0SnMoUEvYqAtxP
  +
J/N3JaiytWlesPPFWvhU/JGUAld5coEU2gbYtlenV/YmdjilIBu50sMSPGF5/6gv
  +
BAA/DbkCDQRetE7mARAA0OH1pn0vdEfSm1kdqIDP3BXBD0BRHNNgGpyXXRRJFaip
  +
bmpu7jSv3FsvN/NmG3BcLXXLFvwY/eIOr6fxRye+a5FSQEtvBnI1GHNmD5GAVT/H
  +
KiwrT5e3ReR/FQS7hCXWU4OA2bKmSEdkJ952NhyYeyAKbkOBgbnlEhtWOAdMI7ws
  +
peHAlHDqfGVOKXDh+FddCUQj/yZ2rblSzFdcC9gtcJSyHWgOQdVAEesEZ16hcZoj
  +
+6O+6BXOQWOo7EPD7lA9a1qesBkSRcxQn48IVVZ2Qx2P2FtCfF+SFX+HQdqJGl15
  +
qxE5CXTuJCMmCVnWhvcLW405uF/HmMFXdqGobEDiQsFFQrfpPVOi4T90VkW8P81s
  +
uPoAlWht1CppNnmhWlvPQsPK/oSMBBOvOEH1EnWJate8yIkveNbqzrE7Xt3sjF6k
  +
yqXaF+qW8OcDvSH/fgvVd21G10Cm77Z2WaKWvfi221oWj+WrgT8cCYv0AVmaLRMe
  +
dajuYlPRQ8KaZaESza2eXggOMP5LQs/mQgfHfwSRekSbKg/L6ctp+xrZ0DPj4iIl
  +
8+H4DxTILopAFWXA1a+uMVp8mV77gA9PyV3nIkrwgaZQ8MdhoKwvN/+SbvhpdzyF
  +
UekzMP/HOaC6JgAomluwnFCdMDFa3FMCF3QUcIyY556QdoFD7g6033xqV6vL+d8A
  +
EQEAAYkCNgQYAQgAIBYhBOigMuCU2OtOoYnScNpBjIijIZ97BQJetE7mAhsMAAoJ
  +
ENpBjIijIZ97lecP+wTgSqhCz3TlUshR8lVrzECueIg3jh3+lY56am9X4MoZ2DAW
  +
IXKjWKVWO55WPYD15A7+TbDyb4zh55m81LxSpV0CSRN4aPuixosWP4d0l+363D2F
  +
oudz+QyvoK5J2sKFPMfhdTgGsEYVO/Zbhus5oNi0kjUTD9U7jHWPS3ilvk/g2F+k
  +
T68lL9+oooleeT+kcBvbKt487JUOwMrkmHqNZdh8qmvMASAuqBcEcqjz96kVEMJY
  +
bhn2skexKfIncoo/btixzJUbnplpDfibFxUHhvWWdwIv4kl3YnrCKKGSDoJcG1mV
  +
sQegK4jWVGrqY8MnCI48iotP18ZxyqOycsZvs2jNmFlKwD9s1mrlr97HZ1MYbLWr
  +
Hq06owH0AzVRM7tzMK7EuHkFLcoa8qh3oijn8O0B7xNOKpTZ2DjajQ/1w8nqmMi5
  +
Z3Wie6ivKng/7p6c6HDrKjoQYc0/fuh1YnL60JG2Arn1OwdBsLDlzPL+Ro5iNwoJ
  +
hZ+stxoZT48iAIWonBsLU11Y+MSwWdN1Eh411HTTunrEs6SafMEhnPi7vvUIZhny
  +
Es0qOM/IUR1I0VtsurSn8aA6Y2Bp73+HuqFLx13/tPKBIUo6D7n/ywUlDCo7wtCw
  +
aSgXPw6uF+0CyLOQ0haf2j6w1OB8ayEGSkTPER5rImCJf3MGw8IECGrErAd+
  +
=emKC
  +
-----END PGP PUBLIC KEY BLOCK-----
  +
</pre>
  +
  +
Moodustada apt source list fail
  +
  +
<pre>
  +
tookohaarvuti # cat /etc/apt/sources.list.d/hashicorp-archive-keyring.list
  +
deb [signed-by=/etc/apt/keyrings/hashicorp-archive-keyring.pub] https://apt.releases.hashicorp.com jammy main
  +
</pre>
  +
  +
<pre>
  +
# apt-get update
  +
# apt-get install terraform
  +
</pre>
  +
  +
Tulemusena on arvutisse lisandunud muuseas
  +
  +
<pre>
  +
tookohaarvuti # dpkg -L terraform
  +
/.
  +
/usr
  +
/usr/bin
  +
/usr/bin/terraform
  +
/usr/share
  +
/usr/share/doc
  +
/usr/share/doc/terraform
  +
/usr/share/doc/terraform/changelog.gz
  +
</pre>
  +
  +
terraform utiliiti tuleks kasutada tavakasutajana.
   
 
===cloudinit template ettevalmistamine===
  
===cloudinit template ettevalmistamine===
89. rida: 175. rida:
 
TODO
  
TODO
   
−
===valmis cloudinit template kasutamine===
 +
===Valmis cloudinit template kasutamine - Debian v. 11===
   
−
proxmox# wget https://cdimage.debian.org/cdimage/openstack/current-10/debian-10-openstack-amd64.qcow2
 +
tookohaarvuti# wget http://cloud.debian.org/images/cloud/bullseye/20220911-1135/debian-11-generic-amd64-20220911-1135.qcow2
  +
  +
Tõenäoliselt on abiks lisada tõmmisesse qemu-guest-agent
  +
  +
<pre>
  +
tookohaarvuti# apt install libguestfs-tools
  +
tookohaarvuti# virt-customize -a debian-11-generic-amd64-20220911-1135.qcow2 --install qemu-guest-agent
  +
[ 0.0] Examining the guest ...
  +
[ 31.1] Setting a random seed
  +
virt-customize: warning: random seed could not be set for this type of guest
  +
[ 31.3] Setting the machine ID in /etc/machine-id
  +
[ 31.3] Installing packages: qemu-guest-agent
  +
[ 84.8] Finishing off
  +
  +
tookohaarvuti# rm -rf /var/tmp/.guestfs-0
  +
</pre>
  +
  +
Tõmmise baasil virtuaalse arvuti template moodustamine, esmalt tekitakse vastavaid tegevusi sooritav skript, ja tookohaarvutis valmistaud .qcow2 tõmmis on kopeeritud proxmox serverisse
   
 
<pre>
  
<pre>
 
proxmox# cat /root/qm-create-9000
 
proxmox# cat /root/qm-create-9000
−
qm create 9000 -name debian-10-template -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "Debian 10 cloud image" -kvm 1 -numa 1
 +
qm create 9000 -name debian-11-cloud-template -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "Debian 11 cloud image" -kvm 1
−
qm importdisk 9000 debian-10-openstack-amd64.qcow2 vgdata
 +
qm importdisk 9000 debian-11-generic-amd64-20220911-1135.qcow2 sn_pve
−
qm set 9000 -scsihw virtio-scsi-pci -virtio0 vgdata:vm-9000-disk-0
 +
qm set 9000 -scsihw virtio-scsi-pci -virtio0 sn_pve:vm-9000-disk-0
 
qm set 9000 -serial0 socket
  
qm set 9000 -serial0 socket
 
qm set 9000 -boot c -bootdisk virtio0
  
qm set 9000 -boot c -bootdisk virtio0
 
qm set 9000 -agent 1
  
qm set 9000 -agent 1
−
qm set 9000 -hotplug disk,network,usb,memory,cpu
  
 
qm set 9000 -vcpus 1
  
qm set 9000 -vcpus 1
−
qm set 9000 -vga qxl
 +
qm set 9000 -ide2 sn_pve:cloudinit
−
qm set 9000 -name debian-10-template
  
−
qm set 9000 -ide2 vgdata:cloudinit
  
 
</pre>
  
</pre>
   
  +
ja moodustamine ise
−
<pre>
  
   
  +
<pre>
 
proxmox# sh qm-create-9000
 
proxmox# sh qm-create-9000
−
importing disk 'debian-10-openstack-amd64.qcow2' to VM 9000 ...
 +
importing disk 'debian-11-generic-amd64-20220911-1135.qcow2' to VM 9000 ...
−
WARNING: dos signature detected on /dev/data/vm-9000-disk-0 at offset 510. Wipe it? [y/n]: [n]
 +
WARNING: dos signature detected on /dev/pve/vm-9000-disk-0 at offset 510. Wipe it? [y/n]: [n]
 
Aborted wiping of dos.
  
Aborted wiping of dos.
 
Logical volume "vm-9000-disk-0" created.
  
Logical volume "vm-9000-disk-0" created.
121. rida: 222. rida:
 
...
  
...
   
−
Successfully imported disk as 'unused0:vgdata:vm-9000-disk-0'
 +
Successfully imported disk as 'unused0:sn_pve:vm-9000-disk-0'
−
update VM 9000: -scsihw virtio-scsi-pci -virtio0 vgdata:vm-9000-disk-0
 +
update VM 9000: -scsihw virtio-scsi-pci -virtio0 pve:vm-9000-disk-0
 
update VM 9000: -serial0 socket
  
update VM 9000: -serial0 socket
 
update VM 9000: -boot c -bootdisk virtio0
  
update VM 9000: -boot c -bootdisk virtio0
128. rida: 229. rida:
 
update VM 9000: -hotplug disk,network,usb,memory,cpu
  
update VM 9000: -hotplug disk,network,usb,memory,cpu
 
update VM 9000: -vcpus 1
  
update VM 9000: -vcpus 1
−
update VM 9000: -vga qxl
 +
update VM 9000: -ide2 sn_pve:cloudinit
−
update VM 9000: -name debian-10-template
  
−
update VM 9000: -ide2 vgdata:cloudinit
  
 
Logical volume "vm-9000-cloudinit" created.
  
Logical volume "vm-9000-cloudinit" created.
−
WARNING: iso9660 signature detected on /dev/data/vm-9000-cloudinit at offset 32769. Wipe it? [y/n]: [n]
 +
WARNING: iso9660 signature detected on /dev/pve/vm-9000-cloudinit at offset 32769. Wipe it? [y/n]: [n]
 
Aborted wiping of iso9660.
  
Aborted wiping of iso9660.
 
1 existing signature left on the device.
  
1 existing signature left on the device.
 
</pre>
  
</pre>
   
  +
Tulemuse kontrollimine
−
===terraform seadistamine ja käivitamine===
  
   
  +
* PVE liideses on näha uus virtuaalne arvuti vmid=9000
−
Terraformi faili sisu võib olla nt selline
  
  +
* cloudinit on kasutuses
  +
  +
===Valmis cloudinit template kasutamine - Ubuntu v. 20.04===
  +
  +
tookohaarvuti# wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
  +
  +
===terraform kasutusjuhu seadistamine ja käivitamine===
  +
  +
Terraform kliendi kasutusjuht asub failisüsteemis ühes kataloogis ja sisaldab
  +
  +
* virtualiseerimise platvormi provider'it
  +
* .tf seadistusfaile
  +
* terraform state andmestikku (nt faili kujul; alternatiiv on sql andmebaasi kujul)
  +
  +
Enne kui midagi sisulist tegema asutakse, tuleks minna selleks mõeldud tühja kataloogi, moodustada seal fail main.tf
   
 
<pre>
  
<pre>
  +
$ mkdir tf/kasutusjuht-1
−
# cat provider-proxmox.tf
 
  +
$ cd tf/kasutusjuht-1
  +
  +
$ cat main.tf
  +
terraform {
  +
required_providers {
  +
proxmox = {
  +
source = "telmate/proxmox"
  +
version = "2.9.11"
  +
}
  +
}
  +
}
  +
</pre>
  +
  +
Seejärel öelda init
  +
  +
<pre>
  +
$ terraform init
  +
</pre>
  +
  +
Tulemusena kopeeritakse internetist alla provider
  +
  +
<pre>
  +
$ find .terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/ -type f
  +
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/README.md
  +
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/terraform-provider-proxmox_v2.9.11
  +
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/LICENSE
  +
</pre>
  +
  +
Seejärel saab terraformi sisuliselt kasutama asuda, nt main.tf faili sisu võib olla selline
  +
  +
<pre>
  +
$ cat provider-proxmox.tf
  +
terraform {
  +
required_providers {
  +
proxmox = {
  +
source = "telmate/proxmox"
  +
version = "2.9.11"
  +
}
  +
}
  +
}
  +
 
provider "proxmox" {
  
provider "proxmox" {
 
pm_parallel = 1
  
pm_parallel = 1
−
pm_tls_insecure = true
 +
pm_tls_insecure = "true"
 
pm_api_url = "https://192.168.110.171:8006/api2/json"
  
pm_api_url = "https://192.168.110.171:8006/api2/json"
 
pm_password = "parool"
  
pm_password = "parool"
156. rida: 311. rida:
 
target_node = "ceph-pm0"
  
target_node = "ceph-pm0"
 
desc = "Debian 10 cloud image"
  
desc = "Debian 10 cloud image"
  +
clone = "debian-11-cloud-template"
−  
−
clone = "debian-cloudinit"
  
 
os_type = "cloud-init"
  
os_type = "cloud-init"
 
cores = "1"
  
cores = "1"
165. rida: 319. rida:
 
scsihw = "virtio-scsi-pci"
  
scsihw = "virtio-scsi-pci"
 
bootdisk = "virtio0"
  
bootdisk = "virtio0"
−
agent = "1"
 +
agent = "1"
   
 
network {
  
network {
−
id = 0
  
 
model = "virtio"
  
model = "virtio"
−
bridge = "vmbr0"
 +
bridge = "vmbr1"
  +
tag = 210
 
}
  
}
   
 
disk {
  
disk {
−
id = 0
 +
size = "2G"
−
size = 2
  
 
type = "virtio"
  
type = "virtio"
−
storage = "vgdata"
 +
storage = "sn_pve"
−
storage_type = "lvm"
  
−
iothread = true
  
 
}
  
}
   
 
lifecycle {
  
lifecycle {
−
ignore_changes = [
 +
ignore_changes = [
−
network,
 +
network,
−
]
 +
]
 
}
  
}
   
196. rida: 347. rida:
   
 
sshkeys = <<EOF
  
sshkeys = <<EOF
−
ssh-rsa ....
+
ssh-ed25519 AAAA....
 
EOF
  
EOF
  +
 
}
  
}
  +
  +
# Output Server IP
  +
output "ip" {
  +
value = "${proxmox_vm_qemu.proxmox_vm[*].ipconfig0}"
  +
}
  +
 
</pre>
  
</pre>
   
 
kus
  
kus
   
  +
* pm_parallel - tundub, et äärmiselt oluline proxmox puhul jätta väärtusega 1 (vastasel korral tekib igasuguseid probleeme, lukustusega jne)
  +
* target node - kui paigaldatakse proxmox klastrisse arvuteid, millisele kontreetsele node'ile paigaldada
 
* count - tekitatavate virtuaalsete arvutite arv
  
* count - tekitatavate virtuaalsete arvutite arv
  +
* ipconfig0 jne - cloud-init sistema
−
* TODO
  
  +
* sshkeys - ssh pub võti paigutatakse /root/.ssh/authorized_keys alla kokkuvõttes (cloud-init sistema abil)
   
 
Kasutamiseks sobib öelda
  
Kasutamiseks sobib öelda
   
  +
$ terraform plan
 
$ terraform apply
  
$ terraform apply
   
250. rida: 412. rida:
 
# terraform plan -out planfile
  
# terraform plan -out planfile
 
# terraform apply --auto-approve
  
# terraform apply --auto-approve
  +
  +
===2026 aasta kevad märkused===
  +
  +
====Terraform====
  +
  +
Väited
  +
  +
* telmate/proxmox asemel on populaarne bpg/proxmox - https://registry.terraform.io/providers/bpg/proxmox/latest/docs
  +
  +
Näidis tf
  +
  +
<pre>
  +
# cat main.tf
  +
terraform {
  +
required_providers {
  +
proxmox = {
  +
source = "bpg/proxmox"
  +
version = "0.100.0"
  +
}
  +
}
  +
}
  +
  +
provider "proxmox" {
  +
endpoint = "https://192.168.10.166:8006/"
  +
api_token = var.proxmox_api_token
  +
insecure = true # Because we're brave (and using self-signed certs)
  +
}
  +
  +
resource "proxmox_virtual_environment_vm" "jenkins_spawn_set_of_four_ci" {
  +
for_each = {
  +
"node-01" = { id = 111011, ip = "192.168.10.111/24" }
  +
"node-02" = { id = 111012, ip = "192.168.10.112/24" }
  +
"node-03" = { id = 111013, ip = "192.168.10.113/24" }
  +
"node-04" = { id = 111014, ip = "192.168.10.114/24" }
  +
}
  +
  +
# count = "4"
  +
name = each.key
  +
vm_id = each.value.id
  +
node_name = "pve-svc-02"
  +
  +
# name = "vm-from-jenkins-set-of-four-${count.index + 1}"
  +
# node_name = "pve-svc-02" # Your PVE node name
  +
# vm_id = 111010 + count.index
  +
  +
clone {
  +
vm_id = 9000 # The ID of your Golden Template
  +
full = true
  +
}
  +
  +
initialization {
  +
datastore_id = "si_zp_data"
  +
  +
ip_config {
  +
ipv4 {
  +
address = each.value.ip
  +
gateway = "192.168.10.254" # Change to your actual gateway
  +
}
  +
}
  +
  +
user_account {
  +
username = "ubuntu"
  +
password = "parool"
  +
# Insert your SSH public key here so you can log in immediately
  +
keys = ["ssh-ed25519 AAAAC3NzaC1lZDI... "]
  +
}
  +
}
  +
  +
cpu {
  +
cores = 2
  +
type = "x86-64-v2-AES"
  +
}
  +
  +
memory {
  +
dedicated = 6144
  +
}
  +
  +
network_device {
  +
bridge = "vmbr0"
  +
vlan_id = 10
  +
}
  +
  +
disk {
  +
datastore_id = "si_zp_data"
  +
interface = "virtio0"
  +
size = 20
  +
}
  +
}
  +
  +
variable "proxmox_api_token" {
  +
type = string
  +
sensitive = true
  +
}
  +
</pre>
  +
  +
kus
  +
  +
* kontrollida bpg versioon
  +
* api_token väärtustatakse keskkonnamuutujana TF_VAR_proxmox_api_token (TF_VAR peab olema alguses, see lahutatakse käigult automaatselt maha)
  +
  +
====Cloud-init====
  +
  +
cloud-init toega ubuntu 24.04 pve template tekitamine
  +
  +
<pre>
  +
root@pve-svc-02:~/20260408# wget https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
  +
root@pve-svc-02:~/20260408# qm create 9000 --name "template-ubuntu-2404" --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
  +
root@pve-svc-02:~/20260408# qm set 9000 --scsihw virtio-scsi-pci --scsi0 si_zp_data:0,import-from=$(pwd)/ubuntu-24.04-server-cloudimg-amd64.img
  +
root@pve-svc-02:~/20260408# qm set 9000 --ide2 si_zp_data:cloudinit
  +
root@pve-svc-02:~/20260408# qm set 9000 --boot order=scsi0 --serial0 socket --vga serial0
  +
root@pve-svc-02:~/20260408# qm template 9000
  +
</pre>
   
 
===Kasulikud lisamaterjalid===
  
===Kasulikud lisamaterjalid===

Viimane redaktsioon: 8. aprill 2026, kell 14:41

Sissejuhatus

Terraform https://www.terraform.io/ ...

Mõisted

  • IaC - infrastructure as code

Tööpõhimõte

     töökohaarvuti 1         tf state arvuti          töökohaarvuti 2

 /usr/local/bin/terraform    postgresql baas      /usr/local/bin/terraform
 
          _____                   _____                    _____
         |     |                 |     |                  |     |
         |     |                 |     |                  |     |
         |_____|                 |_____|                  |_____|
            |                       |                        |
            |                       |                        |
        ----|---------|-------------|------------------------|-----
                      |
                      |
                         
                    ......

                      |
          -----|------|--------------|----------------------|---------------------|------
               |                     |                      |                     |
             __|__                 __|__                  __|__                 __|__
            |     |               |     |                |     |               |     |
            |     |               |     |                |     |               |     |
            |_____|               |_____|                |_____|               |_____|

           proxmox 1             proxmox 2              proxmox 3             proxmox 4

                            https://192.168.10.191, 2, 3, 4:8006/api/

kus

  • proxmox 1, 2, 3 ja 4 on host arvutid, millel töötavad virtuaalsed arvutid
  • töötakoharvutites 1 ja 2 kasutatakse terraform tarkvara
  • terraform state arvutis on salvestatud terraform ettekujutus serveritesse tekitatud virtuaalsete arvutite koosseisust
  • terraform peab arvet nii virtuaalsete arvutite hulga ja asukoha üle kui ka virtuaalsete arvutite sisu üle (virtuaalse riistvara kooseis, ip seadistus jms)

Uus virtuaalne arvuti kujuneb kolme sisendi alusel

  • varem ettevalmistatud cloud-init võimeline proxmox qemu (ehk kvm) template
  • proxmox snippets cloud-init seadistused
  • terraform

Paigaldamine

2020

Terraform tarkvara jagatakse aadressil https://www.terraform.io/downloads.html ühe zipitud binary faili kujul. nt 2020 aasta suvel sobib kopeerida 

# cd /var/tmp
# wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip

Paigaldamiseks tuleb see lahti pakkida 

# unzip terraform_0.12.28_linux_amd64.zip

ja kopeerida nt kataloogi /usr/local/bin 

# cp terraform /usr/local/bin
# chmod 0755 /usr/local/bin/terraform

Tulemusena saab nt küsida tarkvara versiooni 

imre@deb11-tookoht:~$ terraform -v
Terraform v0.12.28

imre@deb11-tookoht:~$ ldd /usr/local/bin/terraform
	not a dynamic executable

imre@deb11-tookoht:~$ file /usr/local/bin/terraform 
/usr/local/bin/terraform: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=oMODVmlDWDtedK13OzTZ/3LrauOon2ma8s4bZsv2i/YIWvbuMZtz346Y44Ol4L/B0x9UGkPWCqOH_nEZK7-, not stripped

2022

Üks juhend asub aadressil https://austinsnerdythings.com/2021/09/01/how-to-deploy-vms-in-proxmox-with-terraform/ ja originaal https://www.terraform.io/cli/install/apt

Kopeerida tootja juurest repo pgp võti 

tookohaarvuti# wget https://apt.releases.hashicorp.com/gpg -O /etc/apt/keyrings/hashicorp-archive-keyring.pub
tookohaarvuti# cat /etc/apt/keyrings/hashicorp-archive-keyring.pub
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBF60TuYBEADLS1MP7XrMlRkn1Y54cb2UclUMH8HkIRfBrhk5Leo9kNZc/2QD
LmdQbi3UbZkz0uVkHqbFDgV5lAnukCnxgr9BqnL0GJpO78le7gCCbM5bR4rTJ6Ar
OOtIKf25smGTIpbSwNdj8BOLqiExGFj/9L5X9S5kfq3vtuYt+lmxKkIrEPjSYnFR
TQ2mTL8RM932GJod/5VJ2+6YvrCjtPu5/rW02H1U2ZHiTtX6ZGnIvv/sprKyFRqT
x4Ib+o9XwXof/LuxTMpVwIHSzCYanH5hPc7yRGKzIntBS+dDom+h9smx7FTgpHwt
QRFGLtVoHXqON6nXTLFDkEzxr+fXq/bgB1Kc1TuzvoK601ztQGhhDaEPloKqNWM8
Ho7JU1RpnoWr5jOFTYiPM9uyCtFNsJmD9mt4K8sQQN7T2inR5Us0o510FqePRFeX
wOJUMi1CbeYqVHfKQ5cWYujcK8pv3l1a6dSBmFfcdxtwIoA16JzCrgsCeumTDvKu
hOiTctb28srL/9WwlijUzZy6R2BGBbhP937f2NbMS/rpby7M1WizKeo2tkKVyK+w
SUWSw6EtFJi7kRSkH7rvy/ysU9I2ma88TyvyOgIz1NRRXYsW7+brgwXnuJraOLaB
5aiuhlngKpTPvP9CFib7AW2QOXustMZ7pOUREmxgS4kqxo74CuFws163TwARAQAB
tFFIYXNoaUNvcnAgU2VjdXJpdHkgKEhhc2hpQ29ycCBQYWNrYWdlIFNpZ25pbmcp
IDxzZWN1cml0eStwYWNrYWdpbmdAaGFzaGljb3JwLmNvbT6JAk4EEwEIADgWIQTo
oDLglNjrTqGJ0nDaQYyIoyGfewUCXrRO5gIbAwULCQgHAgYVCgkICwIEFgIDAQIe
AQIXgAAKCRDaQYyIoyGfe6/WD/9dTM/1OSgbvSPpPJOOcn5L1nOKRBJpztr4V0ky
GoCDakIQ/sykbcuHXP79FGLzrM8zQOsbvVp/Z2lsWBnxkT8KWM+8LZxYToRGdZhr
huFPHV9df0vAsZGisu4ejHDneHOTO3KqVotkky34jUSjBL7Q8uwXHY9r+5hb452N
vafN1w0Y1QVhb6JjjwWHR8Rf9qkSIEi6m9o8a1M54yQC2y/Zrs6+4F3zZ4uYfTvz
MyFfj0P5VmAoaowLSRdb2/JTObu0+zpKN+PjZA8BcnOf/pvqmEz83FIfo6zJLScx
TVaAwj5Iz/jS04x7EvBuIP3vpgv1R6r+t0qU/7hpu7Oc0dsxhL+C8BpVY26/2hvX
ozN5eG0ysSwexqwls+bnRgd6KdoHlWFNfbW8RCPKyb/s+tmFqGAY/QmxMkukgnXQ
WvBoa0Gdv2AFVLYup9tEO1zF4zBPh5oQwAXDNudLTHJ4KmyEwWsOQJUjNB4y4a7j
iGgK77T4KKXpo7pVDP8Ur+tmNH/d+/YFjxrfJvWt4ypE5dZmFO/FrUMvIGglOLDt
A+SiQe73IpEebB8PiqNlqJ2NU7artuRxYQVColt+/1puIHwV+h0SnMoUEvYqAtxP
J/N3JaiytWlesPPFWvhU/JGUAld5coEU2gbYtlenV/YmdjilIBu50sMSPGF5/6gv
BAA/DbkCDQRetE7mARAA0OH1pn0vdEfSm1kdqIDP3BXBD0BRHNNgGpyXXRRJFaip
bmpu7jSv3FsvN/NmG3BcLXXLFvwY/eIOr6fxRye+a5FSQEtvBnI1GHNmD5GAVT/H
KiwrT5e3ReR/FQS7hCXWU4OA2bKmSEdkJ952NhyYeyAKbkOBgbnlEhtWOAdMI7ws
peHAlHDqfGVOKXDh+FddCUQj/yZ2rblSzFdcC9gtcJSyHWgOQdVAEesEZ16hcZoj
+6O+6BXOQWOo7EPD7lA9a1qesBkSRcxQn48IVVZ2Qx2P2FtCfF+SFX+HQdqJGl15
qxE5CXTuJCMmCVnWhvcLW405uF/HmMFXdqGobEDiQsFFQrfpPVOi4T90VkW8P81s
uPoAlWht1CppNnmhWlvPQsPK/oSMBBOvOEH1EnWJate8yIkveNbqzrE7Xt3sjF6k
yqXaF+qW8OcDvSH/fgvVd21G10Cm77Z2WaKWvfi221oWj+WrgT8cCYv0AVmaLRMe
dajuYlPRQ8KaZaESza2eXggOMP5LQs/mQgfHfwSRekSbKg/L6ctp+xrZ0DPj4iIl
8+H4DxTILopAFWXA1a+uMVp8mV77gA9PyV3nIkrwgaZQ8MdhoKwvN/+SbvhpdzyF
UekzMP/HOaC6JgAomluwnFCdMDFa3FMCF3QUcIyY556QdoFD7g6033xqV6vL+d8A
EQEAAYkCNgQYAQgAIBYhBOigMuCU2OtOoYnScNpBjIijIZ97BQJetE7mAhsMAAoJ
ENpBjIijIZ97lecP+wTgSqhCz3TlUshR8lVrzECueIg3jh3+lY56am9X4MoZ2DAW
IXKjWKVWO55WPYD15A7+TbDyb4zh55m81LxSpV0CSRN4aPuixosWP4d0l+363D2F
oudz+QyvoK5J2sKFPMfhdTgGsEYVO/Zbhus5oNi0kjUTD9U7jHWPS3ilvk/g2F+k
T68lL9+oooleeT+kcBvbKt487JUOwMrkmHqNZdh8qmvMASAuqBcEcqjz96kVEMJY
bhn2skexKfIncoo/btixzJUbnplpDfibFxUHhvWWdwIv4kl3YnrCKKGSDoJcG1mV
sQegK4jWVGrqY8MnCI48iotP18ZxyqOycsZvs2jNmFlKwD9s1mrlr97HZ1MYbLWr
Hq06owH0AzVRM7tzMK7EuHkFLcoa8qh3oijn8O0B7xNOKpTZ2DjajQ/1w8nqmMi5
Z3Wie6ivKng/7p6c6HDrKjoQYc0/fuh1YnL60JG2Arn1OwdBsLDlzPL+Ro5iNwoJ
hZ+stxoZT48iAIWonBsLU11Y+MSwWdN1Eh411HTTunrEs6SafMEhnPi7vvUIZhny
Es0qOM/IUR1I0VtsurSn8aA6Y2Bp73+HuqFLx13/tPKBIUo6D7n/ywUlDCo7wtCw
aSgXPw6uF+0CyLOQ0haf2j6w1OB8ayEGSkTPER5rImCJf3MGw8IECGrErAd+
=emKC
-----END PGP PUBLIC KEY BLOCK-----

Moodustada apt source list fail 

tookohaarvuti # cat /etc/apt/sources.list.d/hashicorp-archive-keyring.list 
deb [signed-by=/etc/apt/keyrings/hashicorp-archive-keyring.pub] https://apt.releases.hashicorp.com jammy main

# apt-get update
# apt-get install terraform

Tulemusena on arvutisse lisandunud muuseas 

tookohaarvuti # dpkg -L terraform
/.
/usr
/usr/bin
/usr/bin/terraform
/usr/share
/usr/share/doc
/usr/share/doc/terraform
/usr/share/doc/terraform/changelog.gz

terraform utiliiti tuleks kasutada tavakasutajana.

cloudinit template ettevalmistamine

TODO

Valmis cloudinit template kasutamine - Debian v. 11

tookohaarvuti# wget http://cloud.debian.org/images/cloud/bullseye/20220911-1135/debian-11-generic-amd64-20220911-1135.qcow2

Tõenäoliselt on abiks lisada tõmmisesse qemu-guest-agent 

tookohaarvuti# apt install libguestfs-tools
tookohaarvuti# virt-customize -a debian-11-generic-amd64-20220911-1135.qcow2 --install qemu-guest-agent
[   0.0] Examining the guest ...
[  31.1] Setting a random seed
virt-customize: warning: random seed could not be set for this type of guest
[  31.3] Setting the machine ID in /etc/machine-id
[  31.3] Installing packages: qemu-guest-agent
[  84.8] Finishing off

tookohaarvuti# rm -rf /var/tmp/.guestfs-0

Tõmmise baasil virtuaalse arvuti template moodustamine, esmalt tekitakse vastavaid tegevusi sooritav skript, ja tookohaarvutis valmistaud .qcow2 tõmmis on kopeeritud proxmox serverisse 

proxmox# cat /root/qm-create-9000 
qm create 9000 -name debian-11-cloud-template -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "Debian 11 cloud image" -kvm 1
qm importdisk 9000 debian-11-generic-amd64-20220911-1135.qcow2 sn_pve
qm set 9000 -scsihw virtio-scsi-pci -virtio0 sn_pve:vm-9000-disk-0
qm set 9000 -serial0 socket
qm set 9000 -boot c -bootdisk virtio0
qm set 9000 -agent 1
qm set 9000 -vcpus 1
qm set 9000 -ide2 sn_pve:cloudinit

ja moodustamine ise 

proxmox# sh qm-create-9000 
importing disk 'debian-11-generic-amd64-20220911-1135.qcow2' to VM 9000 ...
WARNING: dos signature detected on /dev/pve/vm-9000-disk-0 at offset 510. Wipe it? [y/n]: [n]
  Aborted wiping of dos.
  Logical volume "vm-9000-disk-0" created.
  1 existing signature left on the device.
transferred: 0 bytes remaining: 2147483648 bytes total: 2147483648 bytes progression: 0.00 %
transferred: 21474836 bytes remaining: 2126008812 bytes total: 2147483648 bytes progression: 1.00 %
transferred: 42949672 bytes remaining: 2104533976 bytes total: 2147483648 bytes progression: 2.00 %
...

Successfully imported disk as 'unused0:sn_pve:vm-9000-disk-0'
update VM 9000: -scsihw virtio-scsi-pci -virtio0 pve:vm-9000-disk-0
update VM 9000: -serial0 socket
update VM 9000: -boot c -bootdisk virtio0
update VM 9000: -agent 1
update VM 9000: -hotplug disk,network,usb,memory,cpu
update VM 9000: -vcpus 1
update VM 9000: -ide2 sn_pve:cloudinit
  Logical volume "vm-9000-cloudinit" created.
WARNING: iso9660 signature detected on /dev/pve/vm-9000-cloudinit at offset 32769. Wipe it? [y/n]: [n]
  Aborted wiping of iso9660.
  1 existing signature left on the device.

Tulemuse kontrollimine

  • PVE liideses on näha uus virtuaalne arvuti vmid=9000
  • cloudinit on kasutuses

Valmis cloudinit template kasutamine - Ubuntu v. 20.04

tookohaarvuti# wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img

terraform kasutusjuhu seadistamine ja käivitamine

Terraform kliendi kasutusjuht asub failisüsteemis ühes kataloogis ja sisaldab

  • virtualiseerimise platvormi provider'it
  • .tf seadistusfaile
  • terraform state andmestikku (nt faili kujul; alternatiiv on sql andmebaasi kujul)

Enne kui midagi sisulist tegema asutakse, tuleks minna selleks mõeldud tühja kataloogi, moodustada seal fail main.tf 

$ mkdir tf/kasutusjuht-1
$ cd tf/kasutusjuht-1

$ cat main.tf
terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "2.9.11"
    }
  }
}

Seejärel öelda init 

$ terraform init

Tulemusena kopeeritakse internetist alla provider 

$ find .terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/ -type f
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/README.md
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/terraform-provider-proxmox_v2.9.11
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/LICENSE

Seejärel saab terraformi sisuliselt kasutama asuda, nt main.tf faili sisu võib olla selline 

$ cat provider-proxmox.tf 
terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "2.9.11"
    }
  }
}

provider "proxmox" {
  pm_parallel       = 1
  pm_tls_insecure   = "true"
  pm_api_url        = "https://192.168.110.171:8006/api2/json"
  pm_password       = "parool"
  pm_user           = "root@pam"
}

resource "proxmox_vm_qemu" "proxmox_vm" {
  count             = "5"
  name              = "tf-vm-${count.index + 1}"
  target_node       = "ceph-pm0"
  desc              = "Debian 10 cloud image"
  clone             = "debian-11-cloud-template"
  os_type           = "cloud-init"
  cores             = "1"
  sockets           = "1"
  cpu               = "kvm64"
  memory            = "2048"
  scsihw            = "virtio-scsi-pci"
  bootdisk          = "virtio0"
  agent             = "1"

  network {
        model = "virtio"
        bridge = "vmbr1"
        tag = 210
  }

  disk {
        size = "2G"
        type = "virtio"
        storage = "sn_pve"
  }

  lifecycle {
        ignore_changes = [
          network,
        ]
  }

#  cicustom         = "user=local:snippets/userconfig-${count.index + 1}.yaml"
  ipconfig0         = "ip=192.168.110.6${count.index + 1 }/24,gw=192.168.110.189"
  nameserver        = "8.8.8.8"
  searchdomain      = "sise.moraal.ee"
  ciuser            = "debian"
  cipassword        = "parool123"

  sshkeys = <<EOF
  ssh-ed25519 AAAA....
  EOF

}

# Output Server IP
output "ip" {
  value = "${proxmox_vm_qemu.proxmox_vm[*].ipconfig0}"
}

kus

  • pm_parallel - tundub, et äärmiselt oluline proxmox puhul jätta väärtusega 1 (vastasel korral tekib igasuguseid probleeme, lukustusega jne)
  • target node - kui paigaldatakse proxmox klastrisse arvuteid, millisele kontreetsele node'ile paigaldada
  • count - tekitatavate virtuaalsete arvutite arv
  • ipconfig0 jne - cloud-init sistema
  • sshkeys - ssh pub võti paigutatakse /root/.ssh/authorized_keys alla kokkuvõttes (cloud-init sistema abil)

Kasutamiseks sobib öelda 

$ terraform plan
$ terraform apply

Tulemusena tekib

  • 5 virtuaalset arvutit
  • terraform.tfstate fail (json formaadis)

Kasutamine - PostgreSQL

# cat provider.tf 
provider "postgresql" {
  host            = "192.168.110.51"
  port            = 5432
  database        = "postgres"
  username        = "postgres"
  password        = "parool"
  sslmode         = "require"
  connect_timeout = 15
}

resource "postgresql_database" "my_db" {
  name              = "my_db"
#  owner             = "my_role"
  template          = "template0"
  lc_collate        = "C"
  connection_limit  = -1
  allow_connections = true
}

resource "postgresql_database" "my_db2" {
  name              = "my_db2"
#  owner             = "my_role"
  template          = "template0"
  lc_collate        = "C"
  connection_limit  = -1
  allow_connections = true
}

# terraform plan -out planfile
# terraform apply --auto-approve

2026 aasta kevad märkused

Terraform

Väited

Näidis tf 

# cat main.tf
terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "0.100.0"
    }
  }
}

provider "proxmox" {
  endpoint = "https://192.168.10.166:8006/"
  api_token = var.proxmox_api_token
  insecure = true # Because we're brave (and using self-signed certs)
}

resource "proxmox_virtual_environment_vm" "jenkins_spawn_set_of_four_ci" {
  for_each = {
    "node-01" = { id = 111011, ip = "192.168.10.111/24" }
    "node-02" = { id = 111012, ip = "192.168.10.112/24" }
    "node-03" = { id = 111013, ip = "192.168.10.113/24" }
    "node-04" = { id = 111014, ip = "192.168.10.114/24" }
  }

  # count = "4"
  name      = each.key
  vm_id     = each.value.id
  node_name = "pve-svc-02"

  # name      = "vm-from-jenkins-set-of-four-${count.index + 1}"
  # node_name = "pve-svc-02" # Your PVE node name
  # vm_id     = 111010 + count.index

  clone {
    vm_id = 9000 # The ID of your Golden Template
    full  = true
  }

  initialization {
    datastore_id = "si_zp_data"

    ip_config {
      ipv4 {
        address = each.value.ip
        gateway = "192.168.10.254" # Change to your actual gateway
      }
    }

    user_account {
      username = "ubuntu"
      password = "parool"
      # Insert your SSH public key here so you can log in immediately
      keys     = ["ssh-ed25519 AAAAC3NzaC1lZDI... "] 
    }
  }

  cpu {
    cores = 2
    type  = "x86-64-v2-AES"
  }

  memory {
    dedicated = 6144
  }

  network_device {
    bridge = "vmbr0"
    vlan_id = 10
  }

  disk {
    datastore_id = "si_zp_data"
    interface    = "virtio0"
    size         = 20
  }
}

variable "proxmox_api_token" {
  type      = string
  sensitive = true
}

kus

  • kontrollida bpg versioon
  • api_token väärtustatakse keskkonnamuutujana TF_VAR_proxmox_api_token (TF_VAR peab olema alguses, see lahutatakse käigult automaatselt maha)

Cloud-init

cloud-init toega ubuntu 24.04 pve template tekitamine 

root@pve-svc-02:~/20260408# wget https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
root@pve-svc-02:~/20260408# qm create 9000 --name "template-ubuntu-2404" --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
root@pve-svc-02:~/20260408# qm set 9000 --scsihw virtio-scsi-pci --scsi0 si_zp_data:0,import-from=$(pwd)/ubuntu-24.04-server-cloudimg-amd64.img
root@pve-svc-02:~/20260408# qm set 9000 --ide2 si_zp_data:cloudinit
root@pve-svc-02:~/20260408# qm set 9000 --boot order=scsi0 --serial0 socket --vga serial0
root@pve-svc-02:~/20260408# qm template 9000

Kasulikud lisamaterjalid

Pärit leheküljelt "https://www.auul.pri.ee/wiki/index.php?title=Terraform&oldid=3118"