[SOLVED] PBS unvalid snapshots

[tonci]

Hello to the team,
1st of all congrats for PBS which makes PVE more pwerfull than it was (also powerfull) before. I've made test-setup (1pve host, PBS as VM, pbs store -> local dir in PBS as nfs mount (netgear RN102) ) I took me a while to figure out how it works/handles "remote" store such as this very slow home-NAS.

The point of this post is (snapshot) verification.
Few days pbs-datastore (NAS) was offline and this produced 3 unvalid incremental backups (attach)

and it's still something happening with them (activity circle) I tried to delete them (garbage-can) but instead of getting rid of them this error message appears. The question is whether this 3 "snapshots" affect this VM backup in general ? Does this last valid/verified inc backup/snapshot mean that everything is OK ?
How can I get rid of this 3 non-verified snapshots ?

thank you very much in advance

BR
Tonci

 

[tonci]

status update :



In pve host I lowered max-backup down to 5 and I suppose this picture shows attempt of keeping only last 5 five snapshots and deleting older ones ... But this lasts for too many days already ...
This would be garbage collection responsibility or ?

BR
Tonci

 

[dcsapak]

can you post the ouptut of

ls -lhaR <path>

? and replace <path> with the path to a backupgroup, e.g. /mnt/datastore/vm/301

 

[tonci]

root@pbs03:~# ls -alhR /mnt/nfs-pbs/store1/vm/301/
/mnt/nfs-pbs/store1/vm/301/:
total 4,0K
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 .
drwxrwxr-x+ 1 backup backup 12 stu 14 22:58 ..
drwxrwxr-x+ 1 backup backup 0 stu 15 09:15 2020-10-26T11:15:04Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-29T10:15:03Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-29T12:07:18Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-29T20:09:12Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-30T19:00:03Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-31T19:00:04Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-11-01T19:00:03Z
drwxrwxr-x+ 1 backup backup 190 stu 4 19:51 2020-11-02T19:00:04Z
drwxrwxr-x+ 1 backup backup 190 stu 4 19:47 2020-11-03T19:00:04Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-11-04T19:00:03Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-11-11T19:59:40Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-11-12T19:00:04Z
drwxrwxr-x+ 1 backup backup 220 stu 14 07:27 2020-11-13T19:38:33Z
drwxrwxr-x+ 1 backup backup 220 stu 15 19:22 2020-11-14T08:55:09Z
drwxrwxr-x+ 1 backup backup 220 stu 15 19:25 2020-11-14T18:48:31Z
-rw-rw-r--+ 1 backup backup 9 lis 26 12:14 owner

'/mnt/nfs-pbs/store1/vm/301/2020-10-26T11:15:04Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 15 09:15 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-29T10:15:03Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-29T12:07:18Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-29T20:09:12Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-30T19:00:03Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-31T19:00:04Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-01T19:00:03Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-02T19:00:04Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 190 stu 4 19:51 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 1,6K stu 2 20:05 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 2 20:02 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 2 20:05 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 578 stu 4 19:51 index.json.blob
-rw-r--r--+ 1 backup backup 336 stu 2 19:59 qemu-server.conf.blob

'/mnt/nfs-pbs/store1/vm/301/2020-11-03T19:00:04Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 190 stu 4 19:47 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 1,8K stu 3 20:07 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 3 20:04 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 3 20:07 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 575 stu 4 19:47 index.json.blob
-rw-r--r--+ 1 backup backup 336 stu 3 19:59 qemu-server.conf.blob

'/mnt/nfs-pbs/store1/vm/301/2020-11-04T19:00:03Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-11T19:59:40Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-12T19:00:04Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-13T19:38:33Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 220 stu 14 07:27 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 2,7K stu 13 23:46 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 13 20:50 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 13 23:46 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 610 stu 14 07:27 index.json.blob
-rw-rw-r--+ 1 backup backup 0 stu 13 23:46 .index.json.lck
-rw-r--r--+ 1 backup backup 336 stu 13 20:38 qemu-server.conf.blob

'/mnt/nfs-pbs/store1/vm/301/2020-11-14T08:55:09Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 220 stu 15 19:22 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 2,4K stu 14 12:58 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 14 10:05 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 14 12:57 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 594 stu 15 19:22 index.json.blob
-rw-rw-r--+ 1 backup backup 0 stu 14 12:57 .index.json.lck
-rw-r--r--+ 1 backup backup 336 stu 14 09:55 qemu-server.conf.blob

'/mnt/nfs-pbs/store1/vm/301/2020-11-14T18:48:31Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 220 stu 15 19:25 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 2,6K stu 14 22:53 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 14 22:53 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 14 22:53 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 593 stu 15 19:25 index.json.blob
-rw-rw-r--+ 1 backup backup 0 stu 14 22:53 .index.json.lck
-rw-r--r--+ 1 backup backup 336 stu 14 19:48 qemu-server.conf.blob
root@pbs03:~#

 

[dcsapak]

did you try again to delete those backups? (it seems the dirs are empty)
what filesystem is mounted there? (i guess nfs)
what are the mount options?

 

[tonci]

fstab -> 10.10.75.10:/data/nfs-pbs /mnt/nfs-pbs nfs rw,async,soft,intr,noexec 0 0


still the same error message ...

 

[tonci]

and it always creates new dirty-bitmaps:
INFO: virtio0: dirty-bitmap status: created new
INFO: virtio1: dirty-bitmap status: created new

 

[tonci]

upgraded to 1.0.3 but still same issues

 

[dcsapak]

fstab -> 10.10.75.10:/data/nfs-pbs /mnt/nfs-pbs nfs rw,async,soft,intr,noexec 0 0

yes there is currently a problem with how we lock backups that does not work on nfs

basically we flock a file in the backupdir -> remove the locked file (this works fine on local fs) -> remove the dir
nfs creates a substitute file in step 2 which leads to a 'ENOTEMPTY' error when removing the directory

we'll see how we can fix this in a backwards compatible manner

and it always creates new dirty-bitmaps:
INFO: virtio0: dirty-bitmap status: created new
INFO: virtio1: dirty-bitmap status: created new

this seems to be a different and unrelated issue.. maybe you want to open a new thread with more infos (e.g. the task logs of the backups)

 

[ShagoY]

Oh my god, I just spent a while trying to get PBs to work properly on an NFS/CIFS share...

Can't wait for the fix

 

[tonci]

Hello, it seems that snapshots are handled correctly now (1.0.6) ... They can be erased manually and automatically as well ...
Thanks for the good job and happy NY2021