pve-zsync sync Error

[yena]

Hello,
i'm tryng to setup a cron job to backup one KVM Virtual Machine using pve-zsync on a local HD.
The first sync is OK but from the second one i have this error:
--------------------------------------------------------------------------------------------------------------------------------
pve-zsync sync --source 100 --dest 127.0.0.1:STORAGE/SNAPSHOT --verbose --maxsnap 7 --name WinSrv2012
send from @ to rpool/KVM/vm-100-disk-1@rep_WinSrv2012_2016-07-11_09:28:33 estimated size is 12.7G
total estimated size is 12.7G
TIME SENT SNAPSHOT
warning: cannot send 'rpool/KVM/vm-100-disk-1@rep_WinSrv2012_2016-07-11_09:28:33': Broken pipe
COMMAND:
zfs send -v -- rpool/KVM/vm-100-disk-1@rep_WinSrv2012_2016-07-11_09:28:33 | ssh root@127.0.0.1 -- zfs recv -F -- STORAGE/SNAPSHOT/vm-100-disk-1
GET ERROR:
cannot receive new filesystem stream: destination has snapshots (eg. STORAGE/SNAPSHOT/vm-100-disk-1@rep_BkWinSrv2012_2016-07-08_12:38:33)
must destroy them to overwrite it

--------------------------------------------------------------------------------------------------------------------------------

My Zfs volumes:
zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
STORAGE 9.16G 2.62T 96K /STORAGE
STORAGE/SNAPSHOT 9.16G 2.62T 96K /STORAGE/SNAPSHOT
STORAGE/SNAPSHOT/vm-100-disk-1 9.16G 2.62T 9.16G -
STORAGE/SNAPSHOT/vm-100-disk-1@rep_BkWinSrv2012_2016-07-08_12:38:33 0 - 9.16G -
rpool 27.0G 1.73T 96K /rpool
rpool/KVM 12.4G 1.73T 96K /rpool/KVM
rpool/KVM/vm-100-disk-1 12.4G 1.73T 10.9G -
rpool/KVM/vm-100-disk-1@rep_BkWinSrv2012_2016-07-08_12:38:33 1.47G - 7.91G -
rpool/LXC 96K 1.73T 96K /rpool/LXC
rpool/ROOT 6.15G 1.73T 96K /rpool/ROOT
rpool/ROOT/pve-1 6.15G 1.73T 6.15G /
rpool/data 96K 1.73T 96K /rpool/data
rpool/swap 8.50G 1.74T 64K -

--------------------------------------------------------------------------------------------------------------------------------

Last pve:
pveversion -v
proxmox-ve: 4.2-56 (running kernel: 4.4.13-1-pve)
pve-manager: 4.2-15 (running version: 4.2-15/6669ad2c)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.13-1-pve: 4.4.13-56
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-42
qemu-server: 4.0-83
pve-firmware: 1.1-8
libpve-common-perl: 4.0-70
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-55
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-19
pve-container: 1.0-70
pve-firewall: 2.0-29
pve-ha-manager: 1.0-32
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5.7-pve10~bpo80

Thanks!

 

[wolfgang]

Hi,

can you send me the version of your pve-zsync packet?
and try without local host.
try to use

pve-zsync sync --source 100 --dest STORAGE/SNAPSHOT --verbose --maxsnap 7 --name WinSrv2012

 

[Gurn_Blanston]

I am having similar problems in the following scenario:

Run a command like this:

pve-zsync create --source 10.0.1.70:104 --dest pve1zpool2/zsync --maxsnap 8 --verbose --name dcfs2 --skip

Then before it runs I doctor up the /etc/cron.d/pve-zsync config file to run on my desired schedule. I have several VMs and having them all try to do an initial sync or even a recurring sync at the same time saturates the storage system. I would appreciate any ideas on how to set the timing in the initial pve-zsync create command.

Then after a while, I realize that something isn't how I want it with the sync job and decide to kill it and start over with a new one. I run the command:

pve-zsync destroy --source 10.0.1.70:100 --dest pve1zpool2/zsync --name mqa10

The job gets commented out or deleted, I forget which, in /etc/cron.d/pve-zsync.

As it says in the pve-zsync guide destroying the job doesn't destroy the synced zvols at the destination. So, I do a

zfs destroy -r pve1zpool2/zsync/vm-100-disk-1

It usually works and the zvol is gone along with all of its snapshots.

So far so good, but now when I create the fresh job with my desired changes but using the same job name, it runs for a few cycles then sends me a cron error every time the job runs complaining that a snapshot that I have deleted can't be found and it just can't go on. Here is an actual message.

WARN: COMMAND:


zfs list -rt snapshot -Ho name pve1zpool2/zsync/vm-100-disk-1@rep_mqa10_2016-08-08_16:28:55


GET ERROR:


cannot open 'pve1zpool2/zsync/vm-100-disk-1@rep_mqa10_2016-08-08_16:28:55': dataset does not exist


COMMAND:


ssh root@10.0.1.70 -- zfs send -- zpool_fast/zfs_128k/vm-100-disk-1@rep_mqa10_2016-08-10_08:30:01 | cstream -t 314572800 | zfs recv -F -- pve1zpool2/zsync/vm-100-disk-1 GET ERROR:


cannot receive new filesystem stream: destination has snapshots (eg. pve1zpool2/zsync/vm-100-disk-1@rep_mqa10_2016-08-09_16:30:01)


must destroy them to overwrite it


Job --source 10.0.1.70:100 --name mqa10 got an ERROR!!!

I destroyed the @rep_mqa10_2016-08-08_16:28:55 snapshot when I destroyed the original job. Why does the new job care about the deleted snapshot/zvol in the first place? Why isn't the counter starting with the creation of the new job? Am I creating this problem by destroying the previous job's snapshots? Is it keeping track of the number of snaps to keep even after I destroy the job? I am using the same job name each time I re-create it. Is the number of snaps tied somehow to the job name, which persists after destroying the job? I just can't seem to make it just run its number of snaps and continue.

My goal is to have it run once per hour so that if we find a problem in a vm at 5 pm, we still have shapshots from 9 am. No matter what I have tried, it runs for a while then starts spamming me with "broken pipe" errors.

Are there more detailed guidelines on fine tuning running jobs? What if, for instance, you want to change the number of snapshots but don't want to start all over? If I do manually destroy the destination zvols is there some way to reset the counters so it isn't expecting non-existent snapshots? Or, am I completely misreading this error?

Thanks,

GB

 

[Gurn_Blanston]

Please excuse the slight inconsistency between commands. I meant for the commands and error to all reflect the same job but one of the commands still refers to job name dcfs2. It should say mqa10.

I just attempted to create a brand new job with a fresh name. Hopefully this time it will work.

 

[Gurn_Blanston]

Update. My new sync job is so far working fine.

I have also just noticed that even though I have destroyed the zsynced snapshots from previous jobs, there still linger snapshots on the sending side. See below.

pve1zpool2/zsync/vm-100-disk-1                                                             35.7G  4.96T  35.7G  
-pve1zpool2/zsync/vm-100-disk-1@rep_mqa10_2016-08-09_16:30:01               0      -  35.7G  
-pve1zpool2/zsync/vm-100-disk-2                                                                          260K  4.96T   260K                          -pve1zpool2/zsync/vm-100-disk-2@rep_mqa10_2016-08-09_16:30:01               0      -   260K  -

I hope this isn't too mangled looking. I can't see these snapshots in the PVE GUI and am a little nervous about destroying them. This is production. I need to merge these snapshots into the vdisks but before I destroy them I need to understand why they aren't visible as snapshots in the Web GUI. So, I think I can conclude that not only do you have to manually destroy the target zvols when you want to redo a sync job, you also have to get rid of any lingering snapshots on the sending side. Yikes.

 

[wolfgang]

I need to merge these snapshots into the vdisks but before I destroy them I need to understand why they aren't visible as snapshots in the Web GUI.

Because the snapshots on GUI is in the VM/CT config.

 

[Gurn_Blanston]

I see. Thanks Wolfgang. So just to be absolutely sure, there is no history file somewhere that Pve-zsync refers to that causes it to expect a certain snapshot from a previous job? Is there a simpler way that is integrated into Pve-zsync to clean up its snapshots when you destroy a job?

Thank you,

GB