[SOLVED] Data recovery after failed synchronization

N

nikko6263

New Member
Feb 28, 2024
3
0
1
Hello. On one of the PBS, remote synchronization with another PBS was launched. As a result, some snapshots were deleted. Is it possible to restore them?
Code: 
2024-06-20T08:58:29+03:00: sync datastore 'pve' start

2024-06-20T08:58:29+03:00: found 76 groups to sync

2024-06-20T08:58:30+03:00: sync group ct/117 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/120 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/122 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/133 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/134 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/135 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/137 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/138 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/139 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/140 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/142 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/143 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/144 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/145 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/146 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/150 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/155 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/156 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/157 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/159 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/163 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/165 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/166 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/169 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/171 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/172 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:30+03:00: sync group ct/175 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/183 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/185 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/186 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/187 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/189 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/192 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/193 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/195 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/196 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/199 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/202 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/203 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/204 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/205 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/206 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/208 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/215 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/218 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/237 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group ct/238 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group vm/105 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group vm/106 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:31+03:00: sync group vm/107 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/108 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/110 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/111 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/113 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/115 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/116 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/118 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/123 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/124 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/125 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/126 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/127 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/128 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/129 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/131 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/156 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/157 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/159 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/173 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/186 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/197 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/201 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/207 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/208 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:32+03:00: sync group vm/212 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:33+03:00: sync group vm/220 failed - owner check failed (root@pam != backup@pbs)

2024-06-20T08:58:33+03:00: delete vanished group 'vm/109'

2024-06-20T08:58:33+03:00: delete vanished group 'vm/100'

2024-06-20T08:58:37+03:00: delete vanished group 'vm/101'

2024-06-20T08:58:39+03:00: delete vanished group 'vm/103'

2024-06-20T08:58:42+03:00: delete vanished group 'vm/102'

2024-06-20T08:58:42+03:00: delete vanished group 'vm/104'

2024-06-20T08:58:45+03:00: delete vanished group 'vm/119'

2024-06-20T08:58:48+03:00: delete vanished group 'vm/112'

2024-06-20T08:58:51+03:00: delete vanished group 'vm/114'

2024-06-20T08:58:53+03:00: delete vanished group 'vm/121'

2024-06-20T08:58:53+03:00: delete vanished group 'vm/130'

2024-06-20T08:58:56+03:00: delete vanished group 'vm/136'

2024-06-20T08:58:56+03:00: delete vanished group 'vm/148'

2024-06-20T08:58:56+03:00: delete vanished group 'vm/149'

2024-06-20T08:58:56+03:00: delete vanished group 'vm/151'

2024-06-20T08:58:56+03:00: delete vanished group 'vm/152'

2024-06-20T08:58:56+03:00: delete vanished group 'vm/153'

2024-06-20T08:58:56+03:00: delete vanished group 'vm/174'

2024-06-20T08:58:56+03:00: delete vanished group 'vm/176'

2024-06-20T08:58:57+03:00: delete vanished group 'vm/177'

2024-06-20T08:59:00+03:00: delete vanished group 'vm/178'

2024-06-20T08:59:02+03:00: delete vanished group 'vm/179'

2024-06-20T08:59:02+03:00: delete vanished group 'vm/180'

2024-06-20T08:59:03+03:00: delete vanished group 'vm/181'

2024-06-20T08:59:06+03:00: delete vanished group 'vm/188'

2024-06-20T08:59:12+03:00: delete vanished group 'vm/165'

2024-06-20T08:59:12+03:00: delete vanished group 'vm/194'

2024-06-20T08:59:12+03:00: delete vanished group 'vm/198'

2024-06-20T08:59:15+03:00: delete vanished group 'vm/214'

2024-06-20T08:59:19+03:00: delete vanished group 'vm/216'

2024-06-20T08:59:22+03:00: delete vanished group 'vm/217'

2024-06-20T08:59:22+03:00: delete vanished group 'vm/219'

2024-06-20T08:59:22+03:00: delete vanished group 'vm/160'

2024-06-20T08:59:23+03:00: unable to acquire lock on snapshot directory "/data/pbs_store/pve/vm/160/2024-06-19T23:45:08Z" - possibly running or in use

2024-06-20T08:59:23+03:00: delete vanished group 'vm/161'

2024-06-20T08:59:27+03:00: delete vanished group 'vm/164'

2024-06-20T08:59:29+03:00: delete vanished group 'vm/167'

2024-06-20T08:59:31+03:00: delete vanished group 'vm/168'

2024-06-20T08:59:32+03:00: delete vanished group 'vm/223'

2024-06-20T08:59:32+03:00: delete vanished group 'vm/224'

2024-06-20T08:59:33+03:00: delete vanished group 'vm/225'

2024-06-20T08:59:33+03:00: delete vanished group 'vm/227'

2024-06-20T08:59:35+03:00: delete vanished group 'vm/228'

2024-06-20T08:59:35+03:00: delete vanished group 'vm/229'

2024-06-20T08:59:37+03:00: delete vanished group 'vm/230'

2024-06-20T08:59:37+03:00: delete vanished group 'vm/231'

2024-06-20T08:59:38+03:00: delete vanished group 'vm/232'

2024-06-20T08:59:40+03:00: delete vanished group 'vm/233'

2024-06-20T08:59:42+03:00: delete vanished group 'vm/234'

2024-06-20T08:59:42+03:00: delete vanished group 'vm/235'

2024-06-20T08:59:43+03:00: delete vanished group 'vm/236'

2024-06-20T08:59:44+03:00: delete vanished group 'ct/131'

2024-06-20T08:59:44+03:00: delete vanished group 'ct/132'

2024-06-20T08:59:46+03:00: delete vanished group 'ct/141'

2024-06-20T08:59:46+03:00: delete vanished group 'ct/147'

2024-06-20T08:59:47+03:00: delete vanished group 'ct/154'

2024-06-20T08:59:49+03:00: delete vanished group 'ct/158'

2024-06-20T08:59:51+03:00: delete vanished group 'ct/160'

2024-06-20T08:59:52+03:00: delete vanished group 'ct/161'

2024-06-20T08:59:53+03:00: delete vanished group 'ct/162'

2024-06-20T08:59:55+03:00: delete vanished group 'ct/164'

2024-06-20T08:59:56+03:00: delete vanished group 'ct/167'

2024-06-20T08:59:57+03:00: delete vanished group 'ct/168'

2024-06-20T08:59:59+03:00: delete vanished group 'ct/170'

2024-06-20T08:59:59+03:00: delete vanished group 'ct/174'

2024-06-20T09:00:00+03:00: delete vanished group 'ct/182'

2024-06-20T09:00:00+03:00: delete vanished group 'ct/184'

2024-06-20T09:00:02+03:00: delete vanished group 'ct/102'

2024-06-20T09:00:03+03:00: delete vanished group 'ct/190'

2024-06-20T09:00:03+03:00: delete vanished group 'ct/191'

2024-06-20T09:00:05+03:00: delete vanished group 'ct/200'

2024-06-20T09:00:05+03:00: delete vanished group 'ct/209'

2024-06-20T09:00:07+03:00: delete vanished group 'ct/210'

2024-06-20T09:00:08+03:00: delete vanished group 'ct/211'

2024-06-20T09:00:09+03:00: delete vanished group 'ct/213'

2024-06-20T09:00:11+03:00: delete vanished group 'ct/176'

2024-06-20T09:00:12+03:00: delete vanished group 'ct/180'

2024-06-20T09:00:14+03:00: delete vanished group 'ct/221'

2024-06-20T09:00:15+03:00: delete vanished group 'ct/222'

2024-06-20T09:00:15+03:00: delete vanished group 'ct/226'

2024-06-20T09:00:16+03:00: delete vanished group 'ct/225'

2024-06-20T09:00:16+03:00: delete vanished group 'ct/239'

2024-06-20T09:00:18+03:00: delete vanished group 'ct/240'

2024-06-20T09:00:18+03:00: TASK ERROR: sync failed with some errors.
 
Last edited:
Hi,
do you have those group still present on another PBS instance? Otherwise I am afraid all snapshots which state delete vanished group are gone. Those with ownership mismatch are save.
 
Chris said:
Hi,
do you have those group still present on another PBS instance? Otherwise I am afraid all snapshots which state delete vanished group are gone. Those with ownership mismatch are save.
Click to expand...
Yes, a similar group is present. The situation is that out of 55 vm only 5 were synchronized, the rest are displayed "delete vanished group"
 
In order to help, please be more specific with what the issue is.

nikko6263 said:
Yes, a similar group is present
Click to expand...
What do you mean by this? Either you have the same groups of snapshots on an other (local or remote) datastore/namespace or the snapshots are gone.

nikko6263 said:
The situation is that out of 55 vm only 5 were synchronized, the rest are displayed "delete vanished group"
Click to expand...
Again, I am not sure I do understand what you mean here. According to the logs you posted, there where 76 groups to sync, but some of them where deleted since vanished on the remote and you had the remove vanished selected in the job. Some groups where not synced, since the owner of the group did not match, this is most likely because you have an incorrect remote selected or an incorrect namespace?

So if you now only see 5 of 55 groups synced, that could be because you have only some of the initial snapshots in the group but not all of them? Or that you have setup a group filter for the sync job?

Please post the full task log of the last sync job, maybe this gives a better understanding.

But in general: If your sync job deleted backup groups on your target PBS instance, then the snapshots contained within this group are deleted. You will not be able to restore them, unless you have another location with the same snapshots. For more details on the sync jobs, see https://pbs.proxmox.com/docs/managing-remotes.html#sync-jobs

Edit: fixed link
 
Last edited:
Chris said:
In order to help, please be more specific with what the issue is.


What do you mean by this? Either you have the same groups of snapshots on an other (local or remote) datastore/namespace or the snapshots are gone.


Again, I am not sure I do understand what you mean here. According to the logs you posted, there where 76 groups to sync, but some of them where deleted since vanished on the remote and you had the remove vanished selected in the job. Some groups where not synced, since the owner of the group did not match, this is most likely because you have an incorrect remote selected or an incorrect namespace?

So if you now only see 5 of 55 groups synced, that could be because you have only some of the initial snapshots in the group but not all of them? Or that you have setup a group filter for the sync job?

Please post the full task log of the last sync job, maybe this gives a better understanding.

But in general: If your sync job deleted backup groups on your target PBS instance, then the snapshots contained within this group are deleted. You will not be able to restore them, unless you have another location with the same snapshots. For more details on the sync jobs, see https://pbs.proxmox.com/docs/managing-remotes.html#sync-jobs

Edit: fixed link
Click to expand...
Thanks for the idea. I checked the mirror and found backups of previously deleted vm and ct.
Thank you for the quick response! My problem is solved
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom