Storage types for HA and live migration

[tuathan]

I have configured a 5 node PVE cluster with VMs accessible via NAT. I created ZFS storage on each PVE node (using some of the hdisks) for VM disks. I then set up VM replication from node 1 to 2, 2 to 3 ... 5 to 1 every 10 mins and using HA could get VMs to migrate and (re-)start automatically (with state up to 10 mins ago) if original node went down but obviously this is not 'live' migration.

For production usage, keeping the VM running (CPU / RAM state) might be more important than VM storage contents (as data will be copied off elsewhere afterwards). Is live migration available using other pooled storage options (e.g. glusterFS or RBD) without the networking requirements of a PVE/CEPH hyperconverge cluster? If so what are the recommended networking/general configuration?

 

[Dominic]

I recently tested live migration on a (virtual) hyper-converged GlusterFS storage and it worked for my test VMs. However, I am not sure about bigger workloads on current Gluster.

 

[tuathan]

I recently tested live migration on a (virtual) hyper-converged GlusterFS storage and it worked for my test VMs. However, I am not sure about bigger workloads on current Gluster.

Thank you. I'll give that a try. Are there any other storage options (https://pve.proxmox.com/wiki/Storage) which could be used for " (virtual) hyper-converged"?

 

[Dominic]

Theoretically you can set a directory storage of Proxmox VE on top of whatever storage system you choose. But I think GlusterFS and Ceph are the first candidates for hyper-converged setups on Proxmox VE. Note that Ceph is better integrated into Proxmox VE and also covered in the subscription agreement.

 

[tuathan]

Thanks. I am very impressed with Proxmox so far. If we end up using it for production I would really like my organisation to get at least a basic subscription to support the great work you guys are doing! Thanks for now.

 

[tuathan]

Theoretically you can set a directory storage of Proxmox VE on top of whatever storage system you choose. But I think GlusterFS and Ceph are the first candidates for hyper-converged setups on Proxmox VE. Note that Ceph is better integrated into Proxmox VE and also covered in the subscription agreement.

I created a GlusterFS cluster across 3 nodes successfully, create a VM on one node and used the GlusterFS for storage. I set the VM as HA with 1 migrate/boot. I simulated a node failure (turn off / not shutdown). The VM migrates to another node but restarts rather than continues so all CPU/RAM state is lost. Is there a way to suspend/preserve CPU/RAM state with this i.e. if there is a failure rather than shutdown?

Am I misunderstanding 'live migration'?

If I turn off the PVE node the VM migrates but reboots (only maintains disk state). If I click on 'migrate' VM (with all PVE nodes remaining on) then the VM migrates and maintains current state (memory /CPU) too.

 

[Dark26]

For the live migration both node need to be alive .

For example if you want to do a maintenance on on node, while running , you move every VM on another node. And then you can reboot the node.

If you have a crash, there is no way to save the content of the ram. So reboot is the only solution.

 

[tuathan]

For the live migration both node need to be alive .

For example if you want to do a maintenance on on node, while running , you move every VM on another node. And then you can reboot the node.

If you have a crash, there is no way to save the content of the ram. So reboot is the only solution.

When you put it like that it makes perfect sense!!!

Is there any software you know of that could run on top of VMs to duplicate a program running on a VM on one physical node to a VM on on another physical node so if one goes down there is redundancy. I have a feeling this is some kind of orchestration...

 

[Dominic]

Depending on what software you run maybe some reverse proxy / load balancing solution?

 

[tuathan]

Depending on what software you run maybe some reverse proxy / load balancing solution?

well my initial plan was to use SLURM sitting on top of Proxmox for job submission/management so I may be able to use that in some way by defining different compute node pools confined to different physical servers