Terraform

Sissejuhatus

Terraform https://www.terraform.io/ ...

Mõisted

• IaC - infrastructure as code

Tööpõhimõte

     töökohaarvuti 1         tf state arvuti          töökohaarvuti 2

 /usr/local/bin/terraform    postgresql baas      /usr/local/bin/terraform
 
          _____                   _____                    _____
         |     |                 |     |                  |     |
         |     |                 |     |                  |     |
         |_____|                 |_____|                  |_____|
            |                       |                        |
            |                       |                        |
        ----|---------|-------------|------------------------|-----
                      |
                      |
                         
                    ......

                      |
          -----|------|--------------|----------------------|---------------------|------
               |                     |                      |                     |
             __|__                 __|__                  __|__                 __|__
            |     |               |     |                |     |               |     |
            |     |               |     |                |     |               |     |
            |_____|               |_____|                |_____|               |_____|

           proxmox 1             proxmox 2              proxmox 3             proxmox 4

                            https://192.168.10.191, 2, 3, 4:8006/api/

kus

• proxmox 1, 2, 3 ja 4 on host arvutid, millel töötavad virtuaalsed arvutid
• töötakoharvutites 1 ja 2 kasutatakse terraform tarkvara
• terraform state arvutis on salvestatud terraform ettekujutus serveritesse tekitatud virtuaalsete arvutite koosseisust
• terraform peab arvet nii virtuaalsete arvutite hulga ja asukoha üle kui ka virtuaalsete arvutite sisu üle (virtuaalse riistvara kooseis, ip seadistus jms)

Uus virtuaalne arvuti kujuneb kolme sisendi alusel

• varem ettevalmistatud cloud-init võimeline proxmox qemu (ehk kvm) template
• proxmox snippets cloud-init seadistused
• terraform

Paigaldamine

2020

Terraform tarkvara jagatakse aadressil https://www.terraform.io/downloads.html ühe zipitud binary faili kujul. nt 2020 aasta suvel sobib kopeerida

# cd /var/tmp
# wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip

Paigaldamiseks tuleb see lahti pakkida

# unzip terraform_0.12.28_linux_amd64.zip

ja kopeerida nt kataloogi /usr/local/bin

# cp terraform /usr/local/bin
# chmod 0755 /usr/local/bin/terraform

Tulemusena saab nt küsida tarkvara versiooni

imre@deb11-tookoht:~$ terraform -v
Terraform v0.12.28

imre@deb11-tookoht:~$ ldd /usr/local/bin/terraform
	not a dynamic executable

imre@deb11-tookoht:~$ file /usr/local/bin/terraform 
/usr/local/bin/terraform: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=oMODVmlDWDtedK13OzTZ/3LrauOon2ma8s4bZsv2i/YIWvbuMZtz346Y44Ol4L/B0x9UGkPWCqOH_nEZK7-, not stripped

2022

Üks juhend asub aadressil https://austinsnerdythings.com/2021/09/01/how-to-deploy-vms-in-proxmox-with-terraform/ ja originaal https://www.terraform.io/cli/install/apt

Kopeerida tootja juurest repo pgp võti

tookohaarvuti# wget https://apt.releases.hashicorp.com/gpg -O /etc/apt/keyrings/hashicorp-archive-keyring.pub
tookohaarvuti# cat /etc/apt/keyrings/hashicorp-archive-keyring.pub
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBF60TuYBEADLS1MP7XrMlRkn1Y54cb2UclUMH8HkIRfBrhk5Leo9kNZc/2QD
LmdQbi3UbZkz0uVkHqbFDgV5lAnukCnxgr9BqnL0GJpO78le7gCCbM5bR4rTJ6Ar
OOtIKf25smGTIpbSwNdj8BOLqiExGFj/9L5X9S5kfq3vtuYt+lmxKkIrEPjSYnFR
TQ2mTL8RM932GJod/5VJ2+6YvrCjtPu5/rW02H1U2ZHiTtX6ZGnIvv/sprKyFRqT
x4Ib+o9XwXof/LuxTMpVwIHSzCYanH5hPc7yRGKzIntBS+dDom+h9smx7FTgpHwt
QRFGLtVoHXqON6nXTLFDkEzxr+fXq/bgB1Kc1TuzvoK601ztQGhhDaEPloKqNWM8
Ho7JU1RpnoWr5jOFTYiPM9uyCtFNsJmD9mt4K8sQQN7T2inR5Us0o510FqePRFeX
wOJUMi1CbeYqVHfKQ5cWYujcK8pv3l1a6dSBmFfcdxtwIoA16JzCrgsCeumTDvKu
hOiTctb28srL/9WwlijUzZy6R2BGBbhP937f2NbMS/rpby7M1WizKeo2tkKVyK+w
SUWSw6EtFJi7kRSkH7rvy/ysU9I2ma88TyvyOgIz1NRRXYsW7+brgwXnuJraOLaB
5aiuhlngKpTPvP9CFib7AW2QOXustMZ7pOUREmxgS4kqxo74CuFws163TwARAQAB
tFFIYXNoaUNvcnAgU2VjdXJpdHkgKEhhc2hpQ29ycCBQYWNrYWdlIFNpZ25pbmcp
IDxzZWN1cml0eStwYWNrYWdpbmdAaGFzaGljb3JwLmNvbT6JAk4EEwEIADgWIQTo
oDLglNjrTqGJ0nDaQYyIoyGfewUCXrRO5gIbAwULCQgHAgYVCgkICwIEFgIDAQIe
AQIXgAAKCRDaQYyIoyGfe6/WD/9dTM/1OSgbvSPpPJOOcn5L1nOKRBJpztr4V0ky
GoCDakIQ/sykbcuHXP79FGLzrM8zQOsbvVp/Z2lsWBnxkT8KWM+8LZxYToRGdZhr
huFPHV9df0vAsZGisu4ejHDneHOTO3KqVotkky34jUSjBL7Q8uwXHY9r+5hb452N
vafN1w0Y1QVhb6JjjwWHR8Rf9qkSIEi6m9o8a1M54yQC2y/Zrs6+4F3zZ4uYfTvz
MyFfj0P5VmAoaowLSRdb2/JTObu0+zpKN+PjZA8BcnOf/pvqmEz83FIfo6zJLScx
TVaAwj5Iz/jS04x7EvBuIP3vpgv1R6r+t0qU/7hpu7Oc0dsxhL+C8BpVY26/2hvX
ozN5eG0ysSwexqwls+bnRgd6KdoHlWFNfbW8RCPKyb/s+tmFqGAY/QmxMkukgnXQ
WvBoa0Gdv2AFVLYup9tEO1zF4zBPh5oQwAXDNudLTHJ4KmyEwWsOQJUjNB4y4a7j
iGgK77T4KKXpo7pVDP8Ur+tmNH/d+/YFjxrfJvWt4ypE5dZmFO/FrUMvIGglOLDt
A+SiQe73IpEebB8PiqNlqJ2NU7artuRxYQVColt+/1puIHwV+h0SnMoUEvYqAtxP
J/N3JaiytWlesPPFWvhU/JGUAld5coEU2gbYtlenV/YmdjilIBu50sMSPGF5/6gv
BAA/DbkCDQRetE7mARAA0OH1pn0vdEfSm1kdqIDP3BXBD0BRHNNgGpyXXRRJFaip
bmpu7jSv3FsvN/NmG3BcLXXLFvwY/eIOr6fxRye+a5FSQEtvBnI1GHNmD5GAVT/H
KiwrT5e3ReR/FQS7hCXWU4OA2bKmSEdkJ952NhyYeyAKbkOBgbnlEhtWOAdMI7ws
peHAlHDqfGVOKXDh+FddCUQj/yZ2rblSzFdcC9gtcJSyHWgOQdVAEesEZ16hcZoj
+6O+6BXOQWOo7EPD7lA9a1qesBkSRcxQn48IVVZ2Qx2P2FtCfF+SFX+HQdqJGl15
qxE5CXTuJCMmCVnWhvcLW405uF/HmMFXdqGobEDiQsFFQrfpPVOi4T90VkW8P81s
uPoAlWht1CppNnmhWlvPQsPK/oSMBBOvOEH1EnWJate8yIkveNbqzrE7Xt3sjF6k
yqXaF+qW8OcDvSH/fgvVd21G10Cm77Z2WaKWvfi221oWj+WrgT8cCYv0AVmaLRMe
dajuYlPRQ8KaZaESza2eXggOMP5LQs/mQgfHfwSRekSbKg/L6ctp+xrZ0DPj4iIl
8+H4DxTILopAFWXA1a+uMVp8mV77gA9PyV3nIkrwgaZQ8MdhoKwvN/+SbvhpdzyF
UekzMP/HOaC6JgAomluwnFCdMDFa3FMCF3QUcIyY556QdoFD7g6033xqV6vL+d8A
EQEAAYkCNgQYAQgAIBYhBOigMuCU2OtOoYnScNpBjIijIZ97BQJetE7mAhsMAAoJ
ENpBjIijIZ97lecP+wTgSqhCz3TlUshR8lVrzECueIg3jh3+lY56am9X4MoZ2DAW
IXKjWKVWO55WPYD15A7+TbDyb4zh55m81LxSpV0CSRN4aPuixosWP4d0l+363D2F
oudz+QyvoK5J2sKFPMfhdTgGsEYVO/Zbhus5oNi0kjUTD9U7jHWPS3ilvk/g2F+k
T68lL9+oooleeT+kcBvbKt487JUOwMrkmHqNZdh8qmvMASAuqBcEcqjz96kVEMJY
bhn2skexKfIncoo/btixzJUbnplpDfibFxUHhvWWdwIv4kl3YnrCKKGSDoJcG1mV
sQegK4jWVGrqY8MnCI48iotP18ZxyqOycsZvs2jNmFlKwD9s1mrlr97HZ1MYbLWr
Hq06owH0AzVRM7tzMK7EuHkFLcoa8qh3oijn8O0B7xNOKpTZ2DjajQ/1w8nqmMi5
Z3Wie6ivKng/7p6c6HDrKjoQYc0/fuh1YnL60JG2Arn1OwdBsLDlzPL+Ro5iNwoJ
hZ+stxoZT48iAIWonBsLU11Y+MSwWdN1Eh411HTTunrEs6SafMEhnPi7vvUIZhny
Es0qOM/IUR1I0VtsurSn8aA6Y2Bp73+HuqFLx13/tPKBIUo6D7n/ywUlDCo7wtCw
aSgXPw6uF+0CyLOQ0haf2j6w1OB8ayEGSkTPER5rImCJf3MGw8IECGrErAd+
=emKC
-----END PGP PUBLIC KEY BLOCK-----

Moodustada apt source list fail

tookohaarvuti # cat /etc/apt/sources.list.d/hashicorp-archive-keyring.list 
deb [signed-by=/etc/apt/keyrings/hashicorp-archive-keyring.pub] https://apt.releases.hashicorp.com jammy main

# apt-get update
# apt-get install terraform

Tulemusena on arvutisse lisandunud muuseas

tookohaarvuti # dpkg -L terraform
/.
/usr
/usr/bin
/usr/bin/terraform
/usr/share
/usr/share/doc
/usr/share/doc/terraform
/usr/share/doc/terraform/changelog.gz

terraform utiliiti tuleks kasutada tavakasutajana.

cloudinit template ettevalmistamine

TODO

Valmis cloudinit template kasutamine - Debian v. 11

tookohaarvuti# wget http://cloud.debian.org/images/cloud/bullseye/20220911-1135/debian-11-generic-amd64-20220911-1135.qcow2

Tõenäoliselt on abiks lisada tõmmisesse qemu-guest-agent

tookohaarvuti# apt install libguestfs-tools
tookohaarvuti# virt-customize -a debian-11-generic-amd64-20220911-1135.qcow2 --install qemu-guest-agent
[   0.0] Examining the guest ...
[  31.1] Setting a random seed
virt-customize: warning: random seed could not be set for this type of guest
[  31.3] Setting the machine ID in /etc/machine-id
[  31.3] Installing packages: qemu-guest-agent
[  84.8] Finishing off

tookohaarvuti# rm -rf /var/tmp/.guestfs-0

Tõmmise baasil virtuaalse arvuti template moodustamine, esmalt tekitakse vastavaid tegevusi sooritav skript, ja tookohaarvutis valmistaud .qcow2 tõmmis on kopeeritud proxmox serverisse

proxmox# cat /root/qm-create-9000 
qm create 9000 -name debian-11-cloud-template -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "Debian 11 cloud image" -kvm 1
qm importdisk 9000 debian-11-generic-amd64-20220911-1135.qcow2 sn_pve
qm set 9000 -scsihw virtio-scsi-pci -virtio0 sn_pve:vm-9000-disk-0
qm set 9000 -serial0 socket
qm set 9000 -boot c -bootdisk virtio0
qm set 9000 -agent 1
qm set 9000 -vcpus 1
qm set 9000 -ide2 sn_pve:cloudinit

ja moodustamine ise

proxmox# sh qm-create-9000 
importing disk 'debian-11-generic-amd64-20220911-1135.qcow2' to VM 9000 ...
WARNING: dos signature detected on /dev/pve/vm-9000-disk-0 at offset 510. Wipe it? [y/n]: [n]
  Aborted wiping of dos.
  Logical volume "vm-9000-disk-0" created.
  1 existing signature left on the device.
transferred: 0 bytes remaining: 2147483648 bytes total: 2147483648 bytes progression: 0.00 %
transferred: 21474836 bytes remaining: 2126008812 bytes total: 2147483648 bytes progression: 1.00 %
transferred: 42949672 bytes remaining: 2104533976 bytes total: 2147483648 bytes progression: 2.00 %
...

Successfully imported disk as 'unused0:sn_pve:vm-9000-disk-0'
update VM 9000: -scsihw virtio-scsi-pci -virtio0 pve:vm-9000-disk-0
update VM 9000: -serial0 socket
update VM 9000: -boot c -bootdisk virtio0
update VM 9000: -agent 1
update VM 9000: -hotplug disk,network,usb,memory,cpu
update VM 9000: -vcpus 1
update VM 9000: -ide2 sn_pve:cloudinit
  Logical volume "vm-9000-cloudinit" created.
WARNING: iso9660 signature detected on /dev/pve/vm-9000-cloudinit at offset 32769. Wipe it? [y/n]: [n]
  Aborted wiping of iso9660.
  1 existing signature left on the device.

Tulemuse kontrollimine

• PVE liideses on näha uus virtuaalne arvuti vmid=9000
• cloudinit on kasutuses

Valmis cloudinit template kasutamine - Ubuntu v. 20.04

tookohaarvuti# wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img

terraform kasutusjuhu seadistamine ja käivitamine

Terraform kliendi kasutusjuht asub failisüsteemis ühes kataloogis ja sisaldab

• virtualiseerimise platvormi provider'it
• .tf seadistusfaile
• terraform state andmestikku (nt faili kujul; alternatiiv on sql andmebaasi kujul)

Enne kui midagi sisulist tegema asutakse, tuleks minna selleks mõeldud tühja kataloogi, moodustada seal fail main.tf

$ mkdir tf/kasutusjuht-1
$ cd tf/kasutusjuht-1

$ cat main.tf
terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "2.9.11"
    }
  }
}

Seejärel öelda init

$ terraform init

Tulemusena kopeeritakse internetist alla provider

$ find .terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/ -type f
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/README.md
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/terraform-provider-proxmox_v2.9.11
.terraform/providers/registry.terraform.io/telmate/proxmox/2.9.11/linux_amd64/LICENSE

Seejärel saab terraformi sisuliselt kasutama asuda, nt main.tf faili sisu võib olla selline

$ cat provider-proxmox.tf 
terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "2.9.11"
    }
  }
}

provider "proxmox" {
  pm_parallel       = 1
  pm_tls_insecure   = "true"
  pm_api_url        = "https://192.168.110.171:8006/api2/json"
  pm_password       = "parool"
  pm_user           = "root@pam"
}

resource "proxmox_vm_qemu" "proxmox_vm" {
  count             = "5"
  name              = "tf-vm-${count.index + 1}"
  target_node       = "ceph-pm0"
  desc              = "Debian 10 cloud image"
  clone             = "debian-11-cloud-template"
  os_type           = "cloud-init"
  cores             = "1"
  sockets           = "1"
  cpu               = "kvm64"
  memory            = "2048"
  scsihw            = "virtio-scsi-pci"
  bootdisk          = "virtio0"
  agent             = "1"

  network {
        model = "virtio"
        bridge = "vmbr1"
        tag = 210
  }

  disk {
        size = "2G"
        type = "virtio"
        storage = "sn_pve"
  }

  lifecycle {
        ignore_changes = [
          network,
        ]
  }

#  cicustom         = "user=local:snippets/userconfig-${count.index + 1}.yaml"
  ipconfig0         = "ip=192.168.110.6${count.index + 1 }/24,gw=192.168.110.189"
  nameserver        = "8.8.8.8"
  searchdomain      = "sise.moraal.ee"
  ciuser            = "debian"
  cipassword        = "parool123"

  sshkeys = <<EOF
  ssh-ed25519 AAAA....
  EOF

}

# Output Server IP
output "ip" {
  value = "${proxmox_vm_qemu.proxmox_vm[*].ipconfig0}"
}

kus

• pm_parallel - tundub, et äärmiselt oluline proxmox puhul jätta väärtusega 1 (vastasel korral tekib igasuguseid probleeme, lukustusega jne)
• target node - kui paigaldatakse proxmox klastrisse arvuteid, millisele kontreetsele node'ile paigaldada
• count - tekitatavate virtuaalsete arvutite arv
• ipconfig0 jne - cloud-init sistema
• sshkeys - ssh pub võti paigutatakse /root/.ssh/authorized_keys alla kokkuvõttes (cloud-init sistema abil)

Kasutamiseks sobib öelda

$ terraform plan
$ terraform apply

Tulemusena tekib

• 5 virtuaalset arvutit
• terraform.tfstate fail (json formaadis)

Kasutamine - PostgreSQL

# cat provider.tf 
provider "postgresql" {
  host            = "192.168.110.51"
  port            = 5432
  database        = "postgres"
  username        = "postgres"
  password        = "parool"
  sslmode         = "require"
  connect_timeout = 15
}

resource "postgresql_database" "my_db" {
  name              = "my_db"
#  owner             = "my_role"
  template          = "template0"
  lc_collate        = "C"
  connection_limit  = -1
  allow_connections = true
}

resource "postgresql_database" "my_db2" {
  name              = "my_db2"
#  owner             = "my_role"
  template          = "template0"
  lc_collate        = "C"
  connection_limit  = -1
  allow_connections = true
}

# terraform plan -out planfile
# terraform apply --auto-approve

2026 aasta kevad märkused

Terraform

Väited

• telmate/proxmox asemel on populaarne bpg/proxmox - https://registry.terraform.io/providers/bpg/proxmox/latest/docs

Näidis tf

# cat main.tf
terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "0.100.0"
    }
  }
}

provider "proxmox" {
  endpoint = "https://192.168.10.166:8006/"
  api_token = var.proxmox_api_token
  insecure = true # Because we're brave (and using self-signed certs)
}

resource "proxmox_virtual_environment_vm" "jenkins_spawn_set_of_four_ci" {
  for_each = {
    "node-01" = { id = 111011, ip = "192.168.10.111/24" }
    "node-02" = { id = 111012, ip = "192.168.10.112/24" }
    "node-03" = { id = 111013, ip = "192.168.10.113/24" }
    "node-04" = { id = 111014, ip = "192.168.10.114/24" }
  }

  # count = "4"
  name      = each.key
  vm_id     = each.value.id
  node_name = "pve-svc-02"

  # name      = "vm-from-jenkins-set-of-four-${count.index + 1}"
  # node_name = "pve-svc-02" # Your PVE node name
  # vm_id     = 111010 + count.index

  clone {
    vm_id = 9000 # The ID of your Golden Template
    full  = true
  }

  initialization {
    datastore_id = "si_zp_data"

    ip_config {
      ipv4 {
        address = each.value.ip
        gateway = "192.168.10.254" # Change to your actual gateway
      }
    }

    user_account {
      username = "ubuntu"
      password = "parool"
      # Insert your SSH public key here so you can log in immediately
      keys     = ["ssh-ed25519 AAAAC3NzaC1lZDI... "] 
    }
  }

  cpu {
    cores = 2
    type  = "x86-64-v2-AES"
  }

  memory {
    dedicated = 6144
  }

  network_device {
    bridge = "vmbr0"
    vlan_id = 10
  }

  disk {
    datastore_id = "si_zp_data"
    interface    = "virtio0"
    size         = 20
  }
}

variable "proxmox_api_token" {
  type      = string
  sensitive = true
}

kus

• kontrollida bpg versioon
• api_token väärtustatakse keskkonnamuutujana TF_VAR_proxmox_api_token (TF_VAR peab olema alguses, see lahutatakse käigult automaatselt maha)

Cloud-init

cloud-init toega ubuntu 24.04 pve template tekitamine

root@pve-svc-02:~/20260408# wget https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img
root@pve-svc-02:~/20260408# qm create 9000 --name "template-ubuntu-2404" --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
root@pve-svc-02:~/20260408# qm set 9000 --scsihw virtio-scsi-pci --scsi0 si_zp_data:0,import-from=$(pwd)/ubuntu-24.04-server-cloudimg-amd64.img
root@pve-svc-02:~/20260408# qm set 9000 --ide2 si_zp_data:cloudinit
root@pve-svc-02:~/20260408# qm set 9000 --boot order=scsi0 --serial0 socket --vga serial0
root@pve-svc-02:~/20260408# qm template 9000

Kasulikud lisamaterjalid

• https://www.baeldung.com/ops/terraform-intro
• https://medium.com/swlh/provision-proxmox-vms-with-terraform-quick-and-easy-5ad1975df1de
• https://bugs.launchpad.net/cloud-init/+bug/1850310
• https://blog.gruntwork.io/terraform-up-running-2nd-edition-early-release-is-now-available-b104fc29783f