VM Storage: Shared storage VS. failover to local drives on nodes in cluster

[michaeljd]

Hello,

I'm in the process of planning out a 3-node cluster for failover and high availability and have a question about where to store the VMs. Assume the following;

All servers are on a 10GB backbone
All servers have SSDs for OS and data drives
All 3 PVE nodes are identical. Same MB/CPU/RAM/local storage
TrueNAS is running bare metal on it's own beefy server and all it's drives are SSDs

As I understand it, one of the features of high availability is that if one server goes offline, it's VMs will automatically start on one of more of the other nodes in the cluster. Is there a benefit (or detriment) to having the VMs stored on shared storage instead of locally on the PVE nodes?

In my case, shared storage would be an NFS share on the TrueNAS server. Having the VMs stored on shared storage would remove the requirement for dedicated data/VM drives on the nodes...that is one benefit I can see. Anything else I'm missing or not considering? Thank you in advance.

 

[bbgeek17]

As I understand it, one of the features of high availability is that if one server goes offline, it's VMs will automatically start on one of more of the other nodes in the cluster.

"Start" is the key word here. The VMs will not be live migrated on complete node failure, but rather restarted. Live migration is only available via manual methods at this time. The QEMU project and Proxmox team are working on getting live migration on failure to a production state.

Is there a benefit (or detriment) to having the VMs stored on shared storage instead of locally on the PVE nodes?

If you dont store your data on shared storage and one of your nodes fails, then it will be impossible to start the VMs elsewhere as the data is not there.
Your options are:
- shared storage, live migration in healthy cluster, restart in faulty cluster
- local storage, long live migration (data needs to be copied) in healthy cluster, no support in faulty cluster
- local storage, replication, time point recovery on failure

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[michaeljd]

Thanks for the detailed response, bbgeek17. I did not know live migration was not an option at this time, but that is not a show stopper for me, thankfully. It would be inconvenient to have my Pi-hole and Plex servers go down for a minute or two, but not the end of the world.

I would've sworn I read somewhere that in a PVE cluster with HA, copies of all VMs are stored on all machines in the cluster and those copies are inactive until they are needed b/c their primary node went down. Am I confusing this with something else or am I just incorrect? Thanks again.

 

[bbgeek17]

I did not know live migration was not an option at this time

Live migration is an option, when you initiate it (manually or programmatically).
Live migration cant be done if node failed because memory state of the VM no longer exists.

PVE cluster with HA, copies of all VMs are stored on all machines in the cluster and those copies are inactive

You may be thinking of Replication, implemented currently via ZFS snapshot mirroring. Thats not part of HA cluster, but rather a separate feature that an admin has to implement and use.
https://pve.proxmox.com/wiki/Storage_Replication

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[michaeljd]

Thanks again, bbgeek17. This is very helpful. I have more reading/planning to do, apparently. I'm not expecting my first cluster to be perfect, but I'd prefer I don't screw it up so badly that I have to wipe/reload all the nodes and start over.