I would like to get technical advise on how to setup Proxmox VE 7 properly on a specific hardware. I am used to using Proxmox, but I have no experience in setting up in HCI or using Software Based Storage like Ceph. I was wondering how do I setup at least 3 nodes, and will add more nodes to probably 10/20 nodes in total using OVH DC located in Singapore region only. At the moment, the only server available is Advance-APAC, in which have limited 1Gbps private bandwidth, and either with 2x NVME disk, or 2x NVME + 2x SATA disk only. My aim is to host probably hundreds of VMs. Right now I am hosting VMs on a single node, which is subject to failure and the resources usage is not evenly distributed. So I was wondering on how to better utilise my nodes by combining it into HCI. Of course if you have better idea, let me know. I would prefer not to use 3 tier DC, plus OVH do not have storage server these days in Singapore region for weeks or months now. So I want to restructure my infrastructure if its possible. My aim is to get rid of single point of failure, and to better utilise my node resources. My current setup is all node is isolated and not in cluster mode.
Proxmox on OVH DC with SDS HCI
- Thread starter bari86
- Start date
Using such servers for CEPH is a no go: you will just have 1Gbps interconnect for all CEPH traffic. Also, if using vRack FO IPs or networks, will use the same nics. It will be pretty slow and potentially make the cluster unstable.
I would set a cluster, always with an uneven number of nodes. Spread Corosync links over both nic's of the server. Reserve bandwidth for Corosync traffic (it's easy to fill a 1gbps link). Use zfs in local disks and setup replication for each VM. Place VMs in servers in a way that if the server running it has any downtime, you can run the VM in another server of your cluster i.e. enough CPU and RAM in "backup" server, as disks are replicated disk space will be provisioned already. For critical VMs, store (some of) their backups in another server that does not belong to the replication pair, just in case.
Using a PBS server could provide another level of resiliency and even replace replication in some scenarios, but if there are no storage class servers in that DC then this is not an option.
I would set a cluster, always with an uneven number of nodes. Spread Corosync links over both nic's of the server. Reserve bandwidth for Corosync traffic (it's easy to fill a 1gbps link). Use zfs in local disks and setup replication for each VM. Place VMs in servers in a way that if the server running it has any downtime, you can run the VM in another server of your cluster i.e. enough CPU and RAM in "backup" server, as disks are replicated disk space will be provisioned already. For critical VMs, store (some of) their backups in another server that does not belong to the replication pair, just in case.
Using a PBS server could provide another level of resiliency and even replace replication in some scenarios, but if there are no storage class servers in that DC then this is not an option.