Reactivate replication with existing synchronized snapshot

[carsten2]

I had a replication of a 2 TB VM running. For some unknown reason the special replication snapshots *__replicate* are gone. But I have old manual non-replication snapshot and was able to synchronize the ZFS dataset to the second server. I then tried to create a *__replicate*" snapshot hoping that the replication service will catch up this snapshot any restart replication on this basis. But altough the replication is stopped, the something deletes the snapshots after sometime and the replication job does not work.

Question: How to reactivate the replication job without having the delete the destination and synchronize 2 TB again?

 

[aaron]

You could try to create a snapshot with a name in the same schema as the replication is creating and sync that one before you try the replication again.

 

[carsten2]

You could try to create a snapshot with a name in the same schema as the replication is creating and sync that one before you try the replication again.

I tried but the snapshot got deleted very quickly. Do I have to halt the replication service (how? which is the service name?).
How does Proxmox find the replication snapshot? Does it use the latest one or is the last snapshot registered anywhere?

 

[aaron]

The pvesr.timer is calling the replication runner, you'd have to disable that one or you could also try to remove the replication job. Without looking at the code and if I remember correctly, it will search for an existing snapshot that is older (timestamp) than the current time.

 

[carsten2]

I stopped pvesr.timer, then created a snapshot '__replicate_101-0_1617281220__' for both disks, used 'zfs send -I ' to delta transfer it to the other location and started pvesr.timer. However, the first thing the replication job does, is deleting the "stale" snapshot. Here is the log:

2021-04-01 15:14:00 101-0: start replication job
2021-04-01 15:14:03 101-0: guest => VM 101, running => 29865
2021-04-01 15:14:03 101-0: volumes => vmdata1zfs:vm-101-disk-1,vmdata1zfs:vm-101-disk-2
2021-04-01 15:14:03 101-0: delete stale replication snapshot '__replicate_101-0_1617281220__' on vmdata1zfs:vm-101-disk-1
2021-04-01 15:14:04 101-0: delete stale replication snapshot '__replicate_101-0_1617281220__' on vmdata1zfs:vm-101-disk-2
2021-04-01 15:14:05 101-0: (remote_prepare_local_job) delete stale replication snapshot '__replicate_101-0_1617281220__' on vmdata1zfs:vm-101-disk-2
2021-04-01 15:14:05 101-0: (remote_prepare_local_job) delete stale replication snapshot '__replicate_101-0_1617281220__' on vmdata1zfs:vm-101-disk-1
2021-04-01 15:14:06 101-0: create snapshot '__replicate_101-0_1617282840__' on vmdata1zfs:vm-101-disk-1
2021-04-01 15:14:06 101-0: create snapshot '__replicate_101-0_1617282840__' on vmdata1zfs:vm-101-disk-2
2021-04-01 15:14:06 101-0: using secure transmission, rate limit: none
2021-04-01 15:14:06 101-0: full sync 'vmdata1zfs:vm-101-disk-1' (__replicate_101-0_1617282840__)
2021-04-01 15:14:07 101-0: volume 'data1/vmdata1/vm-101-disk-1' already exists
2021-04-01 15:14:07 101-0: warning: cannot send 'data1/vmdata1/vm-101-disk-1@__replicate_101-0_1617282840__': Broken pipe
2021-04-01 15:14:07 101-0: command 'zfs send -Rpv -- data1/vmdata1/vm-101-disk-1@__replicate_101-0_1617282840__' failed: exit code 1
2021-04-01 15:14:07 101-0: delete previous replication snapshot '__replicate_101-0_1617282840__' on vmdata1zfs:vm-101-disk-1
2021-04-01 15:14:08 101-0: delete previous replication snapshot '__replicate_101-0_1617282840__' on vmdata1zfs:vm-101-disk-2
2021-04-01 15:14:08 101-0: end replication job with error: command 'set -o pipefail && pvesm export vmdata1zfs:vm-101-disk-1 zfs - -with-snapshots 1 -snapshot __replicate_101-0_1617282840__ | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=hal9034' root@xxxxx -- pvesm import vmdata1zfs:vm-101-disk-1 zfs - -with-snapshots 1 -allow-rename 0' failed: exit code 255