High Availability Cluster with PVE-zsync
- Thread starter renss78
- Start date
I think you can use a zfs sync with appropriate snapshots to keep an approximately up-to-date image for another node, but at best you'd have to revert to the most recent snapshot on failover. I don't think you could really call it "HA" since there would be disruption and loss of data.
I'm currently investigating the pve-zsync tool itself. My main concern is that I want to be able to make the backup copy live in the ZFS storage of the remote server (which is also PVE) so that the recovery step of zfs send|zfs recv on the new node is not necessary. For a large disk VM, it can take hours for that process. I don't think that PVE will care if there are additional disk volumes in the ZFS pool that are not part of the VMs running on that node, but I have to evaluate this.
My own strategy (not having read the zsync code yet) would be basically:
qm snapshot
send zfs differential to remote host
zfs destroy older unneeded snapshots on remote and qm destroysnapshot on local.
Right now I'm using this strategy to back up a VM with 1.5TB of disk space that PVE cannot backup itself over NFS because it takes too long.
I'm currently investigating the pve-zsync tool itself. My main concern is that I want to be able to make the backup copy live in the ZFS storage of the remote server (which is also PVE) so that the recovery step of zfs send|zfs recv on the new node is not necessary. For a large disk VM, it can take hours for that process. I don't think that PVE will care if there are additional disk volumes in the ZFS pool that are not part of the VMs running on that node, but I have to evaluate this.
My own strategy (not having read the zsync code yet) would be basically:
qm snapshot
send zfs differential to remote host
zfs destroy older unneeded snapshots on remote and qm destroysnapshot on local.
Right now I'm using this strategy to back up a VM with 1.5TB of disk space that PVE cannot backup itself over NFS because it takes too long.
Very interesting, i am playing around with zfs sync as well. I made a shell script which does the following:
It is constantly watching the "etc/pve/qemu-server" directory for new files.
When a new VM is created, a configuration file(100.conf) will be created in this map.
The script detects this, and uses the name to do the following:
Create a sync job(ve-zsync sync --source $name --dest 192.168.x.x:x/x --verbose --maxsnap 2 --name $name)
Create a recurring sync job(pve-zsync create --$name --dest 92.168.x.x:x/x --verbose --maxsnap 2 --name $name --limit 512 --skip). Each minute it creates a incremental snapshot.
What i understand of your story is that you use a shared storage solution(NFS). I don't want to use shared storage in my case, simply because it is to slow! The script just creates a initial sync job(so the complete VM will be will be copied) and after that it is constantly sending snapshots(incremental) via a Cronjob/tab to the failover server.
All i want is: When a VM on the main server fails, than the failover server must automatically start up the VM on the failover server(with recent snapshots, max 1 minute)
Maybe we can share contact information and fight this storage battle together
Best regards,
Rens
It is constantly watching the "etc/pve/qemu-server" directory for new files.
When a new VM is created, a configuration file(100.conf) will be created in this map.
The script detects this, and uses the name to do the following:
Create a sync job(ve-zsync sync --source $name --dest 192.168.x.x:x/x --verbose --maxsnap 2 --name $name)
Create a recurring sync job(pve-zsync create --$name --dest 92.168.x.x:x/x --verbose --maxsnap 2 --name $name --limit 512 --skip). Each minute it creates a incremental snapshot.
What i understand of your story is that you use a shared storage solution(NFS). I don't want to use shared storage in my case, simply because it is to slow! The script just creates a initial sync job(so the complete VM will be will be copied) and after that it is constantly sending snapshots(incremental) via a Cronjob/tab to the failover server.
All i want is: When a VM on the main server fails, than the failover server must automatically start up the VM on the failover server(with recent snapshots, max 1 minute)
You can do this with the recurring sync job?
What are zfs send and recv for?
Maybe we can share contact information and fight this storage battle together
Best regards,
Rens
What will may be come in future is a async replica function.
this mean your Vm is sync every e.g. 5 min and when the first sever fails the second will continue with max 5 min data leak.
1 min is to much if you have more the 1 vm that is syncing but it depends of data load on this Vm and the resulting sync traffic
But this is not HA!
this mean your Vm is sync every e.g. 5 min and when the first sever fails the second will continue with max 5 min data leak.
1 min is to much if you have more the 1 vm that is syncing but it depends of data load on this Vm and the resulting sync traffic
But this is not HA!
Yes, i this is exactly what i'm looking for. So the VM's getting automatically mirrored via zsync(which is my script doing atm).
Plus the fact that it is aware when a VM fails(or whatever) so the failover server takes it over.
So you are very sure this is absolutely NOT possible in any way at the moment?
I agree on the 1 minute interval, but it is depending on to much factors (e.g. workload, number of VM's etc..) to say anything about that. But 5 minutes looks like a pretty safe value yes.
A interesting fact(what i noticed during the sync proces) about zsync is that the VM's not syncing at once, all sync jobs are placed in some kind of queue. When the previous syncjob is finished... the other syncjob will be started. I noticed this matter during the following scenario:
I made a VM, my script picks it up and automatically does the thing you can read in my previous post. Than i insert a iso in the VM(on the main server) and start installing Windows 7 or a other big lumpy OS. During this proces(the installation) ProxMox start to sync... my nload went peeking(around 500MBit the sec). BUT it kept syncing for quite a while, i think until the data stream was dropping. So i'm pretty sure the 1 minute interval promise of making snapshots is not ensured. On the other hand, only one syncjob at the time is allowed....so the traffic stays acceptable.
Do you agree with me?
Best regards,
Rens
Plus the fact that it is aware when a VM fails(or whatever) so the failover server takes it over.
So you are very sure this is absolutely NOT possible in any way at the moment?
I agree on the 1 minute interval, but it is depending on to much factors (e.g. workload, number of VM's etc..) to say anything about that. But 5 minutes looks like a pretty safe value yes.
A interesting fact(what i noticed during the sync proces) about zsync is that the VM's not syncing at once, all sync jobs are placed in some kind of queue. When the previous syncjob is finished... the other syncjob will be started. I noticed this matter during the following scenario:
I made a VM, my script picks it up and automatically does the thing you can read in my previous post. Than i insert a iso in the VM(on the main server) and start installing Windows 7 or a other big lumpy OS. During this proces(the installation) ProxMox start to sync... my nload went peeking(around 500MBit the sec). BUT it kept syncing for quite a while, i think until the data stream was dropping. So i'm pretty sure the 1 minute interval promise of making snapshots is not ensured. On the other hand, only one syncjob at the time is allowed....so the traffic stays acceptable.
Do you agree with me?
Best regards,
Rens
I use NFS shared storage as the place to put my vzdump backups. This works great up to about 100GB disk space per VM. Even with this size of VM, I turn off compression on the backup because it takes so much longer (and the destination storage is ultimately ZFS with compression, so there's no need to do it twice). On my large VM which has 1.5TB of disk space on a ZFS storage, the backup process takes > 18 hours which is not useful to me. Thus, I wish to use native ZFS tools to back it up.
The zfs send|zfs recv is how you copy ZFS file systems, and incrementally update the remote copies. This is the facility zsync uses as far as I know. If you read the documentation on the zsync wiki page, it describes how to do a recovery and one of the steps is to send/recv the backup onto the new node. What I would prefer is to skip that step by making the backup copy on the new node's storage directly. Copying the ZFS volumes over my LAN runs at about 80MB/s, so it takes several hours to copy my data for this VM which is just unnecessary extra downtime. It is unclear to me (yet) if this is allowed or if it will confuse PVE to have spare volumes in its ZFS storage location.
The zfs send|zfs recv is how you copy ZFS file systems, and incrementally update the remote copies. This is the facility zsync uses as far as I know. If you read the documentation on the zsync wiki page, it describes how to do a recovery and one of the steps is to send/recv the backup onto the new node. What I would prefer is to skip that step by making the backup copy on the new node's storage directly. Copying the ZFS volumes over my LAN runs at about 80MB/s, so it takes several hours to copy my data for this VM which is just unnecessary extra downtime. It is unclear to me (yet) if this is allowed or if it will confuse PVE to have spare volumes in its ZFS storage location.