Switching to proxmox-boot-tool on systems with no potential ESP (Proxmox VE ISO before 5.4) with Ceph

[yavuz]

Hello,

We have a cluster which we have upgraded to 6.4, everything went smooth, thank you for your continous hard work.

We have been reading the known issues and are aware of the following instructions:
https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool

We have switched to proxmox-boot-tool on all hypervisors except the 3 oldest ones. Unfortunately those were installed with a Proxmox VE ISO before 5.4 and those have a different setup on the disks:

# lsblk -o +FSTYPE
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT                FSTYPE
sda      8:0    0 238.5G  0 disk
├─sda1   8:1    0  1007K  0 part
├─sda2   8:2    0 238.5G  0 part                           zfs_member
└─sda9   8:9    0     8M  0 part
sdb      8:16   0 238.5G  0 disk
├─sdb1   8:17   0  1007K  0 part
├─sdb2   8:18   0 238.5G  0 part                           zfs_member
└─sdb9   8:25   0     8M  0 part

We have Ceph active on our cluster and those 3 hypervisors are actually the Ceph monitors, managers and meta data servers. They also have 6 OSD's each from a total of 70 OSD's.

So what is the best way of switching those machines to proxmox-boot-tool? If a reinstallation should be done, what are the steps to take into consideration and in which order?

Any help is appreciated.

 

[aaron]

I am afraid you will have to reinstall them.

A few things need to be taken into consideration. On the PVE side you will have to remove them from the cluster. The admin guide has a section on that. Please also read the note at the end regarding SSH keys.

About the Ceph stuff: How full is your Ceph cluster? I hope there is plenty of space so that the data of one of those 3 nodes can be easily accommodated on the remaining nodes.

I would proceed as follows:

• Create a MON + MDS + MGR on one of the other nodes for the time of the procedure

For each of the 3 nodes, one by one:

• Set the OSDs to "out".
• Wait until Ceph is done rebalancing the data
• Stop and destroy the OSDs
• Destroy the other Ceph services on that node (MON, MGR, MDS)
• Remove the node from the PVE cluster according to linked guide further above
• Reinstall the node
• Add it to the PVE cluster
• Install Ceph services and OSDs
• Wait for the rebalance to finish
• Rinse and repeat for the next node

Once all 3 nodes are reinstalled and the Ceph cluster is healthy again, you can destroy the temporary MON + MDS + MGR on that other node, so only those 3 nodes will have them installed.

 

[yavuz]

Thank you Aaron, we have successfully finished the reinstallation.