Proxmox VE ZFS replication manager released (pve-zsync)

[pjkenned]

Not sure where to put a suggestion for this @wbumiller and everyone.

It would be significantly more useful to have the ability to specify different time period sets for the snapshots. For example one might want to keep:
Recent snapshots (e.g. every 15 minutes for a day)
Daily snapshots
Weekly snapshots
Monthly snapshots

Similar to how rsync.net takes backups/ snapshots of the datasets.

 

[wolfgang]

Hi
You can set this in the crontab manually.

 

[pjkenned]

Hi @wolfgang I see where you can setup times to snapshot and transfer, but I cannot find where to specify - keep 15 minute snapshots for the last day (so 96 snapshots) and keep the last 7 daily snapshots and keep the last 4 weekly snapshots.

I can see how to snapshot every few minutes, days and etc.

 

[wolfgang]

I do not exactly understand what you wont to achieve?

The time of the sync you can set manually in /etc/cron.d/pve-zsync like an normal cronjob.

The amount of how many snapshots you will keepped you have to calculate and set it with -maxsnap

 

[Q-wulf]

I think he's looking for the following:
Last 15 Minutes -> 96 snapshots kept
Daily snapshots -> 7 snapshots kept
Weekly snapshots -> 4 weekly snapshots kept
Monthly snapshots -> 12 weekly snapshots kept.

I think what is eluding @pikenned
is the fact, that you need to take into account using different name or different backup spaces to accomplish this scheme. and create 4 separate jobs to accomplish this for you. It is the same thing you need to do when using vzdump.

• Examples for Name
  • pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup --verbose --maxsnap 96 --name 15Minutes --limit 512
  • pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup --verbose --maxsnap 7 --name Daily --limit 512
  • pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup --verbose --maxsnap 4 --name Weekly --limit 512
  • pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup --verbose --maxsnap 12 --name Monthly --limit 512
• Example for Backup_space
  • pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup/15Minutes --verbose --maxsnap 96 --name BackupJob --limit 512
  • pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup/Daily --verbose --maxsnap 7 --name BackupJob --limit 512
  • pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup/Weekly --verbose --maxsnap 4 --name BackupJob --limit 512
  • pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup/Monthly --verbose --maxsnap 12 --name BackupJob --limit 512

not aware of a simpler option.

 

[RobFantini]

fyi: '--name' needs to be unique , use something like --name 100-daily . for example:

zfs create tank/pve-zsync
zfs create tank/pve-zsync/Daily
zfs create tank/pve-zsync/Weekly
zfs create tank/pve-zsync/Monthly
zfs create tank/pve-zsync/15Min

# /etc/cron.d/pve-zsync
#
# .--------------- minute (0 - 59)
#/    .----------- hour (0 - 23)
#     |  .-------- day of month (1 - 31)
#     |  |  .----- month (1 - 12) OR jan,feb,mar,apr ...
#     |  |  |  .- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
#     |  |  |  |
#     |  |  |  | user command to be executed
*/15  *  *  *  * root pve-zsync sync --source 10.2.2.41:3992 --dest tank/pve-zsync/15Min     --name 3992-15min     --maxsnap 96 --method ssh
15   23  *  *  * root pve-zsync sync --source 10.2.2.41:3992 --dest tank/pve-zsync/Daily     --name 3992-daily     --maxsnap 7  --method ssh
15   23  *  *  6 root pve-zsync sync --source 10.2.2.41:3992 --dest tank/pve-zsync/Weekly    --name 3992-weekly    --maxsnap 4  --method ssh
15   23  1  *  * root pve-zsync sync --source 10.2.2.41:3992 --dest tank/pve-zsync/Monthly   --name 3992-monthly   --maxsnap 12 --method ssh

 

[pomirou]

We just released the brand new Proxmox VE ZFS replication manager (pve-zsync)!

This CLI tool synchronizes your virtual machine (virtual disks and VM configuration) or directory stored on ZFS between two servers - very useful for backup and replication tasks.

A big Thank-you to our active community for all feedback, testing, bug reporting and patch submissions.

Documentation
http://pve.proxmox.com/wiki/PVE-zsync

Git
https://git.proxmox.com/?p=pve-zsync.git;a=summary

Bugtracker
https://bugzilla.proxmox.com/

__________________
Best regards,


Martin Maurer
Proxmox VE project leader

Hi Martin,

I assume that this will not work for containers.
Is there any similar solution for replicating containers to another node for live replication purposes?

Thanks.

Panayiotis

 

[wolfgang]

Hi,

It will not work with vmid but you can use as workaround the diskpath.

 

[andy77]

Thank you for the feedback, I'll read your link with attention

and thank you for having given my script its original form.

I'll do another script witch will be a kind of nagios plugin to compare de date and hour of the last sync and give alert when its too long... i'll put the script here when and if it's done.

Best regards,

@jeanlau
I know this post is quite old, but did you managed to create the nagios script?

Thx

 

[jeanlau]

Hi Andy77,
I am sorry, I finally haven't developed the script in question, anyway, with the new gui zfs sync, even if I had developed this script, it would be obsolete...
If you are looking forward this kind of information, you will have to find how to interrogate the gui sync process which shows success/failed information in the gui about the syncing... (I suppose the info can be retrieved somewhere)
At least, it would be the place I would start to look, if that helps...
I wish you will find what you need and do not hesitate to post it here, if you find a good solution
Best Regards,

 

[andy77]

Hi, I did not really got the "gui zfs sync" in your sentence?? Do you mean the "replication" feature?

Regards
Andreas

 

[jeanlau]

Hi,

Yes that is exactly what i meant

Best Regards,

 

[andy77]

Well unfortunately the new "Replication" feature has still some errors where replication fails!

See my post:
https://forum.proxmox.com/threads/replication-failed-to-some-node.35684/#post-188445

This is why I am using now pve-zsync instead of replication!