Going from Single Node to 3 Node HA

[BORNXenon]

Hi all, long time lurker, new poster, please be gentle!

I've been running a single node with 8 VMs in production for the best part of 2 years now and happily, it has been rock solid. I also have a second node which I use for test VMs and I also run a Windows 10 VM which I use as my development machine.

Both nodes are Fujitsu RX300 S6 with Intel Xeon E5645 Hexacore, 32GB RAM and 10K SAS drives.

I've been looking into Clusters, HA and CEPH and have a (very) small 3 node test cluster running on Intel i5 workstations, I wasn't expecting steller performance but it has run surprisingly well with a single Windows 10 VM on it to test migrations, failover etc, so well in fact that I now want to run a cluster with HA in production.

I have acquired a third RX300 S6 identical to the existing, some extra disks, 3x LSI 9211-8i HBAs, 3x 10GBE NICs and a 10GBE network switch and some SFP+ cables, in short, everything (I believe) I need to run a 3 node HA cluster. I plan to have 2x 146GB SAS disks in a ZFS RAID1 for Proxmox OS and 6x 300GB SAS disks for CEPH per node.

My question now is how best to proceed with minimum downtime.
I assumed (perhaps wrongly) that I could create a new node on my newly acquired server, install CEPH, etc. restore the VMs to it and bring them online. Then nuke the other 2 nodes, fresh install Proxmox, create a cluster and add all 3 nodes to it, then install CEPH on the 2 new nodes and adjust replication settings.

After further reading this is looking like it is either not possible, or extremely problematic!
Would it be more feasible to maybe backup my Dev VM, nuke and reinstall that node and create a 2 node cluster, then install CEPH, restore and bring online the 8 VMs from the remaining single node, then nuke and reinstall it and add it in to the cluster to finally have a working 3 node HA CEPH cluster?

 

[wolfgang]

Hi,

This is just a row concept of how to do this.
Backups of VM should always be available if something goes wrong.

Theoretically, you can create the cluster without stopping the VM but for configuration and update, you have to reboot nodes.
1.) update all nodes to the latest PVE version.
2.) on the productive system where the most imported VM is, create the cluster.
3.) join the new node to the first one.
4.) the dev node you could remove the config and restore it later, but I guess you get a VMID problem?
What I mean is that the existing VMID of node A and Node Dev must be different to join the cluster.
It this is not the case I would back up and restore the VMs. It is the easiest and fasted way.
5.) Now you can install ceph on all nodes and create the osd in the new node and the dev nodes.
6.) Then you can move the disks from the existing VM to the ceph.

 

[BORNXenon]

Hi Wolfgang,

Thank you for the reply.
The problem I have is that the existing node (Node A) requires reinstallation because it is currently configured on 8 300GB SAS disks in a hardware RAID10, im reluctant to nuke it before I've made absolutely sure I CAN restore the VMs successfully, I've seen seemingly perfect backups not restore for unforseen reasons before and I'm always a bit wary, especially given that this is in effect my entire infrastructure!

A potential option I guess is:
1. Configure the new node, (it'd have to be in JBOD to make sure I have capacity)
2. Restore the VMs from backup
3. Create the cluster
4. Nuke the existing nodes and reinstall
5. Join them to the cluster
6. Install ceph on all nodes
7. Create the OSDs on the 2 freshly joined nodes
8. Move the VM disks into CEPH
9. Reconfigure the storage on the first node and add the OSDs.

Do you think this would work in theory?

 

[BORNXenon]

Ok, so I have managed to do this and it has gone relatively smoothly.
For those interested, this is how I did it.

The first existing node (FEn) was running on 8x 300GB SAS disks in a hardware RAID10 configuration and contained 8 VMs.
The second existing node (SEn) was running 3x 146GB SAS disks and 1x 300GB SAS disk in a hardware RAID10 configuration and contained 1 VM.

I purchased a third server (Ts) with 8x 300GB SAS disks.
I also purchased the following:
3 more 146GB SAS disk and 1 more 300GB SAS disk giving me 6x 146GB disks and 18x 300GB SAS disks in total.
3x LSI 9200-8i HBA in IT mode
3x Fujitsu 10GBe dual port network cards
1x 10GBe switch and SFP+ cables
Enough RAM to upgrade each server to at least 64GB

1. installed Proxmox VE on Ts on 2x 146GB disks in ZFS Mirror (node 1)
2. Backed up the sole VM on SEn, tested I could restore it, then nuked the node and reinstalled Proxmox VE on 2x 146GB disks in ZFS Mirror (node 2)
3. Installed Proxmox VE on an old workstation on 2x 500GB disks in ZFS Mirror (node 0)
4. Created Cluster on node 1 and added node 2 and node 0
5. Added 5x 300GB SAS disks to node 1 and node 2
6. Installed and configured CEPH on node 1 and node 2
7. Backed up all VMs on FEn and restored them to the new Cluster
8. Nuked FEn, reinstalled Proxmox VE on 2x 146GB disks in ZFS Mirror (node 3)
9. Added node 3 to the cluster
10. Added an extra 300GB disk to node 1 and node 2 and added 6x 300GB disks to node 3
11. Installed CEPH on node 3
12. Waited for the rebuild and rebalance to complete

Everything seems to be running ok and aside from the sole Windows Server VM having a bit of a slowdown this morning for reasons as yet unknown, its been pretty good performance wise.