Ceph haldamine - cehpadm: erinevus redaktsioonide vahel
Allikas: Imre kasutab arvutit
Mine navigeerimisribaleMine otsikasti
| 519. rida: | 519. rida: | ||
<pre> |
<pre> |
||
| + | root@morint-cf-01:/# ceph orch apply osd --all-available-devices |
||
| + | ... |
||
| + | |||
root@morint-cf-01:/# ceph osd tree |
root@morint-cf-01:/# ceph osd tree |
||
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF |
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF |
||
Redaktsioon: 23. jaanuar 2023, kell 01:17
Sissejuhatus
Aadressil https://docs.ceph.com/en/quincy/install/ kirjeldatakse mitmeid võimalusi ceph lahenduse haldamiseks, tundub, et 2022 aasta kevadel on üks populaarsemaid nn cephadm.
Tööpõhimõte
cephadm lahendusele on iseloomulik
- haldus toimub cephadm utiliidi abi
- ceph komponendid töötavad konteinerites (nt docker); konteinerite pidamise platvorm ei ole cephadm koosseisus, aga sealt edasi on (tõmmised, konteinerid, võrk jne)
- konteinerid põhinevad centos v. 8 operatsioonisüsteemil
root@ca-1:/# cat /etc/os-release NAME="CentOS Stream" VERSION="8" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" PRETTY_NAME="CentOS Stream 8" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:8" HOME_URL="https://centos.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" root@ca-1:/# rpm -qa | wc -l 423
- mgr komponent on oluline süsteemi osa, nt saab ja isegi on eelistatud ceph ressursside haldus läbi nn Dashboard webgui liidese
- Kõik ceph juhtub konteinerites, st põhimõtteliselt ei ole host arvutites mingit ceph tarkvara (v.a. cephadm)
Ettavalmistamine
Kasutamiseks sobib nt Ubuntu 22.04 operatsioonisüsteem, kus peab olema
- docker
# apt-get install docker.io
- ssh võtmete kopeerimiseks on vajalik ssh serverisse logida sisse root kasutajana (kindlasti on ka muid võimalusi selle haldamiseks)
- dns võiks töötada klastri osaliste vahel (nt tekitada kõigile kõigi kohta sissekanded /etc/hosts faili)
- kõigil arvutitel on lisaks kasutamata plokkseade /dev/vdb
- kellaaeg peab olema süngis
Paigaldamine - 3 node klaster
Ubuntu 22.04 keskkonnas sobib öelda cephadm paigaldamiseks
# apt-get install cephadm
Tulemusena paigaldatakse arvutisse sellised failid
root@ca-1:~# dpkg -L cephadm /. /usr /usr/sbin /usr/sbin/cephadm /usr/share /usr/share/doc /usr/share/doc/cephadm /usr/share/doc/cephadm/changelog.Debian.gz /usr/share/doc/cephadm/copyright /usr/share/man /usr/share/man/man8 /usr/share/man/man8/cephadm.8.gz
Klastri moodustavad kolm arvutit, monitor ja osd komponent on kõigil
- ca-1 - 192.168.110.241
- ca-2 - 192.168.110.242
- ca-3 - 192.168.110.243
Klastri tekitamine
root@ca-1:~# cephadm bootstrap --mon-ip 192.168.110.241 Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit systemd-timesyncd.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit systemd-timesyncd.service is enabled and running Host looks OK Cluster fsid: 08507a78-e04a-11ec-8a3c-d32d38dc4891 Verifying IP 192.168.110.241 port 3300 ... Verifying IP 192.168.110.241 port 6789 ... Mon IP `192.168.110.241` is in CIDR network `192.168.110.0/24` - internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling container image quay.io/ceph/ceph:v17... Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network to 192.168.110.0/24 Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 9283 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to /etc/ceph/ceph.pub Adding key to root@localhost authorized_keys... Adding host ca-1... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying prometheus service with default placement... Deploying grafana service with default placement... Deploying node-exporter service with default placement... Deploying alertmanager service with default placement... Enabling the dashboard module... Waiting for the mgr to restart... Waiting for mgr epoch 9... mgr epoch 9 is available Generating a dashboard self-signed certificate... Creating initial admin user... Fetching dashboard port number... Ceph Dashboard is now available at: URL: https://ca-1.sise.moraal.ee:8443/ User: admin Password: 9np4tnlle8 Enabling client.admin keyring and conf on hosts with "admin" label Enabling autotune for osd_memory_target You can access the Ceph CLI with: sudo /usr/sbin/cephadm shell --fsid 08507a78-e04a-11ec-8a3c-d32d38dc4891 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/master/mgr/telemetry/ Bootstrap complete.
Tulemusena on üks moodustunud ceph klaster mille koosseisus on üks monitor jms node
root@ca-1:~# cephadm shell -- ceph -s
Kahe järgneva node lisamiseks sobib öelda
root@ca-1:~# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ca-2 root@ca-1:~# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ca-3 root@ca-1:~# cephadm shell -- ceph orch host add ca-2 192.168.110.242 _admin root@ca-1:~# cephadm shell -- ceph orch host add ca-3 192.168.110.243 _admin
Tulemusena on kolme node klaster kusjuures puuduvad osd komponendid
root@ca-1:~# cephadm shell -- ceph -s
Inferring fsid 08507a78-e04a-11ec-8a3c-d32d38dc4891
Using recent ceph image quay.io/ceph/ceph@sha256:629fda3151cdca8163e2b8b47af23c69b41b829689d9e085bbd2539c046b0eac
cluster:
id: 08507a78-e04a-11ec-8a3c-d32d38dc4891
health: HEALTH_WARN
mon ca-1 is low on available space
OSD count 0 < osd_pool_default_size 3
services:
mon: 3 daemons, quorum ca-1,ca-2,ca-3 (age 3m)
mgr: ca-1.zsonmh(active, since 45m), standbys: ca-2.uaqloy
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
OSD lisamiseks sobib öelda
root@ca-1:~# cephadm shell -- ceph orch daemon add osd ca-1:/dev/vdb root@ca-1:~# cephadm shell -- ceph orch daemon add osd ca-2:/dev/vdb root@ca-1:~# cephadm shell -- ceph orch daemon add osd ca-3:/dev/vdb
Tulemusena on
root@ca-1:~# cephadm shell -- ceph orch device ls
Inferring fsid 08507a78-e04a-11ec-8a3c-d32d38dc4891
Using recent ceph image quay.io/ceph/ceph@sha256:629fda3151cdca8163e2b8b47af23c69b41b829689d9e085bbd2539c046b0eac
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS
ca-1 /dev/vdb hdd 42.9G Insufficient space (<10 extents) on vgs, LVM detected, locked
ca-2 /dev/vdb hdd 42.9G Insufficient space (<10 extents) on vgs, LVM detected, locked
ca-3 /dev/vdb hdd 42.9G Insufficient space (<10 extents) on vgs, LVM detected, locked
root@ca-1:~# cephadm shell -- ceph -s
Inferring fsid 08507a78-e04a-11ec-8a3c-d32d38dc4891
Using recent ceph image quay.io/ceph/ceph@sha256:629fda3151cdca8163e2b8b47af23c69b41b829689d9e085bbd2539c046b0eac
cluster:
id: 08507a78-e04a-11ec-8a3c-d32d38dc4891
health: HEALTH_OK
services:
mon: 3 daemons, quorum ca-1,ca-2,ca-3 (age 10m)
mgr: ca-1.zsonmh(active, since 52m), standbys: ca-2.uaqloy
osd: 3 osds: 3 up (since 83s), 3 in (since 104s)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 61 MiB used, 120 GiB / 120 GiB avail
pgs: 1 active+clean
Tulemusena on ceph klaster kasutatav
- haldusliidses - https://192.168.110.241:8443/
- TODO
Misc haldamise käsud
# cephadm shell
kus saab seejärel nö kasutada käsku ceph, nt
cs # ceph -s
cluster:
id: f2c7bfa6-de94-11ec-9ce3-dd734d1a236b
health: HEALTH_WARN
5 daemons have recently crashed
services:
mon: 5 daemons, quorum ca-0,ca-1,ca-2,ca-3,ca-4 (age 22h)
mgr: ca-0.snatqq(active, since 28h), standbys: ca-1.bhhbmr
osd: 4 osds: 4 up (since 26h), 4 in (since 27h)
rgw: 2 daemons active (2 hosts, 1 zones)
rgw-nfs: 1 daemon active (1 hosts, 1 zones)
data:
pools: 10 pools, 289 pgs
objects: 5.65k objects, 14 GiB
usage: 44 GiB used, 116 GiB / 160 GiB avail
pgs: 289 active+clean
io:
client: 170 B/s rd, 0 op/s rd, 0 op/s wr
Node nimekirja esitamine
root@ca-0:/# ceph orch host ls HOST ADDR LABELS STATUS ca-0 192.168.110.240 _admin ca-1 192.168.110.241 osd ca-2 192.168.110.242 osd ca-3 192.168.110.243 osd ca-4 192.168.110.244 osd 5 hosts in cluster
Node eemaldamine
TODO
OSD nimekirja esitamine
root@ca-0:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.15637 root default -3 0.03909 host ca-1 0 hdd 0.03909 osd.0 up 1.00000 1.00000 -5 0.03909 host ca-2 1 hdd 0.03909 osd.1 up 1.00000 1.00000 -7 0.03909 host ca-3 2 hdd 0.03909 osd.2 up 1.00000 1.00000 -9 0.03909 host ca-4 3 hdd 0.03909 osd.3 up 1.00000 1.00000
OSD eemaldamine
root@ca-0:/# ceph orch osd rm 3 Scheduled OSD(s) for removal root@ca-0:/# ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 3 ca-4 draining 36 False False False 2022-05-29 19:14:10.549604
Kui status ei näita, et midagi on pooleli (samuti webgui ja 'ceph status') sobib öelda
root@ca-0:/# ceph orch device zap ca-4 /dev/vdb --force zap successful for /dev/vdb on ca-4
Tulemusena ca-4 arvutis ei ole enam ceph storage jaoks lvm vg. Seejärel sobib öelda
root@ca-0:/# ceph orch host rm ca-4 Error EINVAL: Not allowed to remove ca-4 from cluster. The following daemons are running in the host: type id -------------------- --------------- crash ca-4 node-exporter ca-4 mon ca-4 Please run 'ceph orch host drain ca-4' to remove daemons from host root@ca-0:/# ceph orch host drain ca-4 Scheduled to remove the following daemons from host 'ca-4' type id -------------------- --------------- crash ca-4 node-exporter ca-4 mon ca-4
Võib olla veel abistada 'ceph orch apply mon "ca-0,ca-1,ca-2,ca-3"' käsuga
root@ca-0:/# ceph orch host rm ca-4 Removed host 'ca-4'
Tulemusena on konternerite protsessid kadunud ca-4 node pealt ja nimekiri
root@ca-0:/# ceph orch host ls HOST ADDR LABELS STATUS ca-0 192.168.110.240 _admin ca-1 192.168.110.241 osd ca-2 192.168.110.242 osd ca-3 192.168.110.243 osd 4 hosts in cluster
Monitoride komplekti määratlemine, ja tulemuse kontrollimine 'ceph -s' abil
root@ca-0:/# ceph orch apply mon "ca-0,ca-1,ca-2,ca-3" Scheduled mon update...
Dokcer konteinerite nimekirja küsimine, igas arvutis
root@ca-0:~# docker ps
OSD ja seejuures host lisamine
root@ca-0:/# ceph orch host add ca-4 Added host 'ca-4' with addr '192.168.110.244' root@ca-0:/# ceph orch daemon add osd ca-4:/dev/vdb Created osd(s) 3 on host 'ca-4'
Misc käsud
root@ca-0:/# ceph orch device ls HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS ca-1 /dev/vdb hdd 42.9G Insufficient space (<10 extents) on vgs, LVM detected, locked ca-2 /dev/vdb hdd 42.9G Insufficient space (<10 extents) on vgs, LVM detected, locked ca-3 /dev/vdb hdd 42.9G Insufficient space (<10 extents) on vgs, LVM detected, locked ca-4 /dev/vdb hdd 42.9G Insufficient space (<10 extents) on vgs, LVM detected, locked
ja
root@ca-0:/# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 2m ago 4h count:1 crash 5/5 9m ago 4h * grafana ?:3000 1/1 2m ago 4h count:1 mgr 2/2 9m ago 4h count:2 mon 4/4 9m ago 5m ca-0;ca-1;ca-2;ca-3 nfs.nfs_service 1/1 2m ago 1h ca-0;ca-1;ca-2;count:1 node-exporter ?:9100 5/5 9m ago 4h * osd 4 9m ago - <unmanaged> prometheus ?:9095 1/1 2m ago 4h count:1 rgw.rgw_essa ?:80 2/2 9m ago 1h count:2
Label lisamine
root@ca-0:/# ceph orch host label add ca-4 osd Added label osd to host ca-4
teises arvutis töötavate konteinerite nimekirja esitamine
root@ca-1:~# cephadm shell -- ceph orch ps ca-3 Inferring fsid 08507a78-e04a-11ec-8a3c-d32d38dc4891 Using recent ceph image quay.io/ceph/ceph@sha256:629fda3151cdca8163e2b8b47af23c69b41b829689d9e085bbd2539c046b0eac NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID crash.ca-3 ca-3 running (3h) 4m ago 3h 6779k - 17.2.0 132aa60d26c9 cfdd110da6f2 mon.ca-3 ca-3 running (3h) 4m ago 3h 87.3M 2048M 17.2.0 132aa60d26c9 ca5a9dc838a9 node-exporter.ca-3 ca-3 *:9100 running (3h) 4m ago 3h 9.94M - 1dbe0e931976 90075af697dd osd.2 ca-3 running (3h) 4m ago 3h 61.1M 4096M 17.2.0 132aa60d26c9 dd62531349b2
Klastri eemaldamine, igas hostis peab olema cephadm programm ja peab ütlema nii
# cephadm rm-cluster --force --zap-osds --fsid f2c7bfa6-de94-11ec-9ce3-dd734d1a236b
Haldamine - S3
Teenuse ettevalmistamiseks sobib öelda
root@ca-0:/# ceph orch apply rgw rgw_essa
Tulemusena tekitatakse webgui liideses
Object Gateway -> Daemons -> ...
ja seejärel webgui liideses
- tekitada kasutaja
Object Gateway -> Users
- tekitada bucket
Object Gateway -> Buckets ...
Kasutamiseks sobib klient arvutis öelda nt
# apt-get install s3fs # tail -n 1 /etc/passwd-s3fs bucketessa:COQIQ5AMGYK4RFBFN41C:3f3TLyaelVQxBnS7SGw7xRsRiqFtAJLKGXYSwKef # s3fs bucketessa /mnt/root -o passwd_file=/etc/passwd-s3fs -o url=http://192.168.110.240/ -o use_path_request_style
Haldamine - NFS
Tundub, et NFS sobib RO ligipääsu tekitamiseks Object Gateway kraamile, avada webgui liideses
NFS -> ...
kus
- Cluster - nfs_service
- Storage Backend - Object Gateway
- Bucket - bucketessa
- Pseudo - /bucketessa
- Access Type - RO
- Squash - no_root_squash
- Transport Protocol - TCP, UDP
- Clients - Any client could connect
Kasutamiseks sobib öelda lihtsal juhul
# mount 192.168.110.240:/bucketessa /mnt/bucketessa/ root@ria-etcd1:~# df -h | tail -n 2 s3fs 256T 0 256T 0% /mnt/root 192.168.110.240:/bucketessa 160G 44G 117G 28% /mnt/bucketessa
Misc
TODO
- prometheus - http://192.168.110.240:9095/alerts
- dashboard - https://192.168.110.240:8443/
- graphana - https://192.168.110.240:3000/
2023 alguse märkmed - crush rule kasutamine
Tundub, et ceph töötab nüüd kõige õigemasti docker-põhiselt.
TODO
Olgu kasutada selline komplekt salvestusseadmeid - ssd ja hdd tüüpi
root@morint-cf-01:/# ceph orch apply osd --all-available-devices ... root@morint-cf-01:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.71988 root default -7 0.23996 host morint-cf-01 1 hdd 0.04880 osd.1 up 1.00000 1.00000 5 hdd 0.04880 osd.5 up 1.00000 1.00000 14 hdd 0.04880 osd.14 up 1.00000 1.00000 8 ssd 0.03119 osd.8 up 1.00000 1.00000 11 ssd 0.03119 osd.11 up 1.00000 1.00000 17 ssd 0.03119 osd.17 up 1.00000 1.00000 -10 0.23996 host morint-cf-02 2 hdd 0.04880 osd.2 up 1.00000 1.00000 4 hdd 0.04880 osd.4 up 1.00000 1.00000 13 hdd 0.04880 osd.13 up 1.00000 1.00000 7 ssd 0.03119 osd.7 up 1.00000 1.00000 10 ssd 0.03119 osd.10 up 1.00000 1.00000 16 ssd 0.03119 osd.16 up 1.00000 1.00000 -3 0.23996 host morint-cf-03 0 hdd 0.04880 osd.0 up 1.00000 1.00000 3 hdd 0.04880 osd.3 up 1.00000 1.00000 12 hdd 0.04880 osd.12 up 1.00000 1.00000 6 ssd 0.03119 osd.6 up 1.00000 1.00000 9 ssd 0.03119 osd.9 up 1.00000 1.00000 15 ssd 0.03119 osd.15 up 1.00000 1.00000
crush reeglite moodustamiseks sobib öelda
root@morint-cf-01:/# ceph osd crush rule create-replicated replicated_hdd default host hdd root@morint-cf-01:/# ceph osd crush rule create-replicated replicated_ssd default host ssd
Nendes tingimustes saab moodustada poolisid nt dashboardil (määrates replication reegli abil millist tüüpi kettaid pool kasutab). Seejuures saab käigult pool all kettad ära vahetada käsuga (kuigi see on natuke loomuvastane, et pool_ssd hakkab asuma hdd tüüpi ketastel)
root@morint-cf-01:/# ceph osd pool set pool_ssd crush_rule replicated_hdd
Seejärel on Dashboardil näha Recovery andmete liikumine ja mõne aja pärast on ssd kettad tühjad ja vastavad andmed on käigult liigutaud hdd ketastele. Samuti käsu väljundist
root@morint-cf-01:/# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -6 ssd 0.28070 root default~ssd -9 ssd 0.09357 host morint-cf-01~ssd 8 ssd 0.03119 osd.8 11 ssd 0.03119 osd.11 17 ssd 0.03119 osd.17 -12 ssd 0.09357 host morint-cf-02~ssd 7 ssd 0.03119 osd.7 10 ssd 0.03119 osd.10 16 ssd 0.03119 osd.16 -5 ssd 0.09357 host morint-cf-03~ssd 6 ssd 0.03119 osd.6 9 ssd 0.03119 osd.9 15 ssd 0.03119 osd.15 -2 hdd 0.43918 root default~hdd -8 hdd 0.14639 host morint-cf-01~hdd 1 hdd 0.04880 osd.1 5 hdd 0.04880 osd.5 14 hdd 0.04880 osd.14 -11 hdd 0.14639 host morint-cf-02~hdd 2 hdd 0.04880 osd.2 4 hdd 0.04880 osd.4 13 hdd 0.04880 osd.13 -4 hdd 0.14639 host morint-cf-03~hdd 0 hdd 0.04880 osd.0 3 hdd 0.04880 osd.3 12 hdd 0.04880 osd.12 -1 0.71988 root default -7 0.23996 host morint-cf-01 1 hdd 0.04880 osd.1 5 hdd 0.04880 osd.5 14 hdd 0.04880 osd.14 8 ssd 0.03119 osd.8 11 ssd 0.03119 osd.11 17 ssd 0.03119 osd.17 -10 0.23996 host morint-cf-02 2 hdd 0.04880 osd.2 4 hdd 0.04880 osd.4 13 hdd 0.04880 osd.13 7 ssd 0.03119 osd.7 10 ssd 0.03119 osd.10 16 ssd 0.03119 osd.16 -3 0.23996 host morint-cf-03 0 hdd 0.04880 osd.0 3 hdd 0.04880 osd.3 12 hdd 0.04880 osd.12 6 ssd 0.03119 osd.6 9 ssd 0.03119 osd.9 15 ssd 0.03119 osd.15
Kasulikud lisamaterjalid
- https://docs.ceph.com/en/quincy/rados/operations/crush-map/
- https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_device_classes
- https://stackoverflow.com/questions/58060333/ceph-how-to-place-a-pool-on-specific-osd
2023 alguse märkmed - hdd pool ees ssd cache tier kasutamine
TODO
Kasulikud lisamaterjalid
- https://danielpersson.dev/2022/08/01/all-you-need-to-know-about-the-crush-map/
- https://docs.ceph.com/en/quincy/dev/cache-pool/
