I am trying to find the configuration file which pvedaemon or rbdmap use to create the list of monitors used for Ceph RBD storage.
I originally set up a PVE cluster with Ceph RBD storage in 2016, this had 2 hybrid compute/storage (172.x.x.101 & 172.x.x.102) nodes and a temporary (172.x.x.199) node to make an odd number for quorum purposes.
I've added to this with additional hybrid compute/storage nodes (172.x.x.103, 104 & 105) over the years, removing the temporary quorum member (.199) early on.
I am recently replacing the earliest nodes with new hardware, after removing 101 & 102 however I ran into an issue where my RBD storage became inaccessible, breaking backups and any VMs which were restarted.
I traced this to pvedaemon which is running the following service pointing to 3 now non-existant ceph monitors:
I was able to recover by creating a new monitor on an upgraded server numbered as 172.x.x.101 but obviously I'd rather that my rbd daemon was pointing at valid monitors (.199 being non-existent for ~ 5 years).
Can anyone point me towards the config for the pvedaemon or rbdmap list of monitors?
Thanks,
Barry
I originally set up a PVE cluster with Ceph RBD storage in 2016, this had 2 hybrid compute/storage (172.x.x.101 & 172.x.x.102) nodes and a temporary (172.x.x.199) node to make an odd number for quorum purposes.
I've added to this with additional hybrid compute/storage nodes (172.x.x.103, 104 & 105) over the years, removing the temporary quorum member (.199) early on.
I am recently replacing the earliest nodes with new hardware, after removing 101 & 102 however I ran into an issue where my RBD storage became inaccessible, breaking backups and any VMs which were restarted.
I traced this to pvedaemon which is running the following service pointing to 3 now non-existant ceph monitors:
/usr/bin/rbd -p sdfpool -m 172.x.x.101,172.x.x.102,172.x.x.199 --auth_supported cephx -n client.admin --keyring /etc/pve/priv/ceph/sdfpoolRBD.keyring ls -l --format json
I was able to recover by creating a new monitor on an upgraded server numbered as 172.x.x.101 but obviously I'd rather that my rbd daemon was pointing at valid monitors (.199 being non-existent for ~ 5 years).
Can anyone point me towards the config for the pvedaemon or rbdmap list of monitors?
Thanks,
Barry