[SOLVED] PBS unvalid snapshots

T

tonci

Renowned Member
Jun 26, 2010
107
8
83
Hello to the team,
1st of all congrats for PBS which makes PVE more pwerfull than it was (also powerfull) before. I've made test-setup (1pve host, PBS as VM, pbs store -> local dir in PBS as nfs mount (netgear RN102) ) I took me a while to figure out how it works/handles "remote" store such as this very slow home-NAS.

The point of this post is (snapshot) verification.
Few days pbs-datastore (NAS) was offline and this produced 3 unvalid incremental backups (attach)
1605354427492.png
and it's still something happening with them (activity circle) I tried to delete them (garbage-can) but instead of getting rid of them this error message appears. The question is whether this 3 "snapshots" affect this VM backup in general ? Does this last valid/verified inc backup/snapshot mean that everything is OK ?
How can I get rid of this 3 non-verified snapshots ?

thank you very much in advance

BR
Tonci
 
status update :

1605519155392.png

In pve host I lowered max-backup down to 5 and I suppose this picture shows attempt of keeping only last 5 five snapshots and deleting older ones ... But this lasts for too many days already ...
This would be garbage collection responsibility or ?

BR
Tonci
 
can you post the ouptut of

ls -lhaR <path>

? and replace <path> with the path to a backupgroup, e.g. /mnt/datastore/vm/301
 
root@pbs03:~# ls -alhR /mnt/nfs-pbs/store1/vm/301/
/mnt/nfs-pbs/store1/vm/301/:
total 4,0K
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 .
drwxrwxr-x+ 1 backup backup 12 stu 14 22:58 ..
drwxrwxr-x+ 1 backup backup 0 stu 15 09:15 2020-10-26T11:15:04Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-29T10:15:03Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-29T12:07:18Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-29T20:09:12Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-30T19:00:03Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-10-31T19:00:04Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-11-01T19:00:03Z
drwxrwxr-x+ 1 backup backup 190 stu 4 19:51 2020-11-02T19:00:04Z
drwxrwxr-x+ 1 backup backup 190 stu 4 19:47 2020-11-03T19:00:04Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-11-04T19:00:03Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-11-11T19:59:40Z
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 2020-11-12T19:00:04Z
drwxrwxr-x+ 1 backup backup 220 stu 14 07:27 2020-11-13T19:38:33Z
drwxrwxr-x+ 1 backup backup 220 stu 15 19:22 2020-11-14T08:55:09Z
drwxrwxr-x+ 1 backup backup 220 stu 15 19:25 2020-11-14T18:48:31Z
-rw-rw-r--+ 1 backup backup 9 lis 26 12:14 owner

'/mnt/nfs-pbs/store1/vm/301/2020-10-26T11:15:04Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 15 09:15 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-29T10:15:03Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-29T12:07:18Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-29T20:09:12Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-30T19:00:03Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-10-31T19:00:04Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-01T19:00:03Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-02T19:00:04Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 190 stu 4 19:51 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 1,6K stu 2 20:05 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 2 20:02 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 2 20:05 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 578 stu 4 19:51 index.json.blob
-rw-r--r--+ 1 backup backup 336 stu 2 19:59 qemu-server.conf.blob

'/mnt/nfs-pbs/store1/vm/301/2020-11-03T19:00:04Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 190 stu 4 19:47 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 1,8K stu 3 20:07 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 3 20:04 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 3 20:07 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 575 stu 4 19:47 index.json.blob
-rw-r--r--+ 1 backup backup 336 stu 3 19:59 qemu-server.conf.blob

'/mnt/nfs-pbs/store1/vm/301/2020-11-04T19:00:03Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-11T19:59:40Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-12T19:00:04Z':
total 0
drwxrwxr-x+ 1 backup backup 0 stu 14 22:53 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..

'/mnt/nfs-pbs/store1/vm/301/2020-11-13T19:38:33Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 220 stu 14 07:27 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 2,7K stu 13 23:46 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 13 20:50 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 13 23:46 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 610 stu 14 07:27 index.json.blob
-rw-rw-r--+ 1 backup backup 0 stu 13 23:46 .index.json.lck
-rw-r--r--+ 1 backup backup 336 stu 13 20:38 qemu-server.conf.blob

'/mnt/nfs-pbs/store1/vm/301/2020-11-14T08:55:09Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 220 stu 15 19:22 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 2,4K stu 14 12:58 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 14 10:05 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 14 12:57 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 594 stu 15 19:22 index.json.blob
-rw-rw-r--+ 1 backup backup 0 stu 14 12:57 .index.json.lck
-rw-r--r--+ 1 backup backup 336 stu 14 09:55 qemu-server.conf.blob

'/mnt/nfs-pbs/store1/vm/301/2020-11-14T18:48:31Z':
total 5,9M
drwxrwxr-x+ 1 backup backup 220 stu 15 19:25 .
drwxrwxr-x+ 1 backup backup 610 stu 14 19:48 ..
-rw-r--r--+ 1 backup backup 2,6K stu 14 22:53 client.log.blob
-rw-rw-r--+ 1 backup backup 404K stu 14 22:53 drive-virtio0.img.fidx
-rw-rw-r--+ 1 backup backup 5,5M stu 14 22:53 drive-virtio1.img.fidx
-rw-r--r--+ 1 backup backup 593 stu 15 19:25 index.json.blob
-rw-rw-r--+ 1 backup backup 0 stu 14 22:53 .index.json.lck
-rw-r--r--+ 1 backup backup 336 stu 14 19:48 qemu-server.conf.blob
root@pbs03:~#
 
did you try again to delete those backups? (it seems the dirs are empty)
what filesystem is mounted there? (i guess nfs)
what are the mount options?
 
fstab -> 10.10.75.10:/data/nfs-pbs /mnt/nfs-pbs nfs rw,async,soft,intr,noexec 0 0

1605564984189.png
still the same error message ...
 
and it always creates new dirty-bitmaps:
INFO: virtio0: dirty-bitmap status: created new
INFO: virtio1: dirty-bitmap status: created new
 
tonci said:
fstab -> 10.10.75.10:/data/nfs-pbs /mnt/nfs-pbs nfs rw,async,soft,intr,noexec 0 0
Click to expand...
yes there is currently a problem with how we lock backups that does not work on nfs

basically we flock a file in the backupdir -> remove the locked file (this works fine on local fs) -> remove the dir
nfs creates a substitute file in step 2 which leads to a 'ENOTEMPTY' error when removing the directory

we'll see how we can fix this in a backwards compatible manner


tonci said:
and it always creates new dirty-bitmaps:
INFO: virtio0: dirty-bitmap status: created new
INFO: virtio1: dirty-bitmap status: created new
Click to expand...
this seems to be a different and unrelated issue.. maybe you want to open a new thread with more infos (e.g. the task logs of the backups)
 
Oh my god, I just spent a while trying to get PBs to work properly on an NFS/CIFS share...

Can't wait for the fix :)
 
Hello, it seems that snapshots are handled correctly now (1.0.6) ... They can be erased manually and automatically as well ...
Thanks for the good job and happy NY2021
 
You must log in or register to reply here.
Top Bottom