Delaying VM's until ceph/cephFS is mounted

[dlasher]

In larger clusters, it can be quite a few seconds until all the OSD's are happy and cephfs is able to mount. Just restarted one cluster today (power loss) and noticed that while all the KVM's started fine, any LXC that used a CEPHFS bind-mount wouldn't start until cephfs was ready. (got unable to mount messages until it was alive)

Thinking that adding:

ConditionPathIsReadWrite=/mnt/pve/cephfs
ConditionDirectoryNotEmpty=/mnt/pve/cephfs

to

lrwxrwxrwx 1 root root 38 Jul 25  2021 pve-guests.service -> /lib/systemd/system/pve-guests.service

should take care of that dependency. Posting in case it helps others, or in case pmx guys have a better suggestion

 

[gurubert]

We found that adding an override file to the pvestatd.service is sufficient as it will add the correct dependencies.
pve-guests.service is already waiting for pvestatd.service and with this pvestatd.service will wait for pve-storage.target which includes the Ceph services and remote-fs.target.

Put these two lines into /etc/systemd/system/pvestatd.service.d/override.conf or edit the unit file with systemctl edit pvestatd.service:

[Unit]
After=pve-storage.target

Your conditions could also be added to pve-guests.service but I really recommend it doing with systemctl edit.

 

[dlasher]

We found that adding an override file to the pvestatd.service is sufficient as it will add the correct dependencies.
pve-guests.service is already waiting for pvestatd.service and with this pvestatd.service will wait for pve-storage.target which includes the Ceph services and remote-fs.target.

Put these two lines into /etc/systemd/system/pvestatd.service.d/override.conf or edit the unit file with systemctl edit pvestatd.service:

[Unit]
After=pve-storage.target

Your conditions could also be added to pve-guests.service but I really recommend it doing with systemctl edit.

Fantastic, thank you - will give it a shot - The option I suggested above seems to work 80% of the time, but leaves at least one node with no started services.

 

[blackpaw]

We found that adding an override file to the pvestatd.service is sufficient as it will add the correct dependencies.
pve-guests.service is already waiting for pvestatd.service and with this pvestatd.service will wait for pve-storage.target which includes the Ceph services and remote-fs.target.

Put these two lines into /etc/systemd/system/pvestatd.service.d/override.conf or edit the unit file with systemctl edit pvestatd.service:

[Unit]
After=pve-storage.target

Your conditions could also be added to pve-guests.service but I really recommend it doing with systemctl edit.

Not working for me - lxc tries to start immediately, before cephfs is ready and fails.

 

[sw-omit]

Not working for me - lxc tries to start immediately, before cephfs is ready and fails.

If you know how long it takes for things to always boot (and add 5 more second buffer), you could also just "simply" go to the GUI <datacenter> --> <node> --> System --> Options --> Start on boot delay --> Configure a delay. If you are in a cluster you do have to configure it per-node, but they don't all have to be the system (of one system is quiker then the other, you can set a lower delay)

 

[blackpaw]

If you know how long it takes for things to always boot (and add 5 more second buffer), you could also just "simply" go to the GUI <datacenter> --> <node> --> System --> Options --> Start on boot delay --> Configure a delay

Yah, not as neat and tidy but does work, which is what matters. Hopefully, its not something that happens often