Deduplication not correctly shown

[JSEHV]

System running: PBS 2.2-1 / Linux proxpbs 5.15.35-1-pve #1 SMP PVE 5.15.35-3

Deduplication not correctly shown (storage in PBS is supplied via NFS).

After the garbage collect run has done it's work, it shows a deduplication factor of 11.61



But when you look the summary of the datastore, it shows incorrectly still 1.00



Also in the stats of the datastore itself, it doesn't show the latest garbage collect statistics, but still 1.00

 

[dcsapak]

can you post the output of

proxmox-backup-debug api get /status/datastore-usage --output-format json

?

 

[JSEHV]

I've attached the json output to this reply in a plain text file.

 

[dcsapak]

ok the data is there, but it's empty/ all zeros...

can you post the content of the file .gc-status in the datastore?
for example if you datastore path is /mnt/datastore/foo:

cat /mnt/datastore/foo/.gc-status

 

[JSEHV]

There seem to be no .gc-status file in the datastore.
Only .lock and .chunks

I haven't manually adjusted anything, just installed PBS and upgraded over time.

Results from the prompt

root@proxpbs:/# cd mnt/backup-pbs/
root@proxpbs:/mnt/backup-pbs# ls -la
total 8279
drwxr-xr-x     5 1001 1001     6 May 23 21:33 .
drwxr-xr-x     4 root root  4096 Jul 14  2021 ..
drwxr-x--- 65538 1001 1001 65538 Jul 14  2021 .chunks
-rw-r--r--     1 1001 1001     0 Jul 14  2021 .lock
drwxr-xr-x     2 1001 1001     2 May 21 18:12 ns
drwxr-xr-x    18 1001 1001    18 May 22 13:16 vm
root@proxpbs:/mnt/backup-pbs# cd ns
root@proxpbs:/mnt/backup-pbs/ns# ls -la
total 1
drwxr-xr-x 2 1001 1001 2 May 21 18:12 .
drwxr-xr-x 5 1001 1001 6 May 23 21:33 ..
root@proxpbs:/mnt/backup-pbs/ns# cd ..
root@proxpbs:/mnt/backup-pbs# cd vm
root@proxpbs:/mnt/backup-pbs/vm# ls -la
total 145
drwxr-xr-x 18 1001 1001 18 May 22 13:16 .
drwxr-xr-x  5 1001 1001  6 May 23 21:33 ..
drwxr-xr-x 13 1001 1001 14 May 25 03:00 111
drwxr-xr-x 13 1001 1001 14 May 25 03:03 112
drwxr-xr-x 13 1001 1001 14 May 25 03:06 113
drwxr-xr-x 13 1001 1001 14 May 25 03:08 114
drwxr-xr-x 13 1001 1001 14 May 25 03:00 201
drwxr-xr-x 13 1001 1001 14 May 25 03:01 202
drwxr-xr-x 13 1001 1001 14 May 25 03:02 203
drwxr-xr-x 13 1001 1001 14 May 25 03:02 204
drwxr-xr-x 13 1001 1001 14 May 25 03:02 205
drwxr-xr-x 13 1001 1001 14 May 25 03:03 208
drwxr-xr-x 13 1001 1001 14 May 25 03:03 209
drwxr-xr-x 13 1001 1001 14 May 25 03:00 321
drwxr-xr-x 13 1001 1001 14 May 25 03:00 322
drwxr-xr-x 13 1001 1001 14 May 25 03:04 900
drwxr-xr-x 13 1001 1001 14 May 25 03:04 901
drwxr-xr-x 13 1001 1001 14 May 25 03:04 902
root@proxpbs:/mnt/backup-pbs/vm#

Upgrade path after initial installation

root@proxpbs:/var/log# zgrep "install proxmox-backup-server" /var/log/dpkg.log.10.gz
2021-07-14 12:47:18 install proxmox-backup-server:amd64 <none> 2.0.4-1

root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.10.gz
2021-07-22 08:48:22 upgrade proxmox-backup-server:amd64 2.0.4-1 2.0.5-2
2021-07-26 12:20:33 upgrade proxmox-backup-server:amd64 2.0.5-2 2.0.7-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.9.gz
2021-08-09 11:33:35 upgrade proxmox-backup-server:amd64 2.0.7-1 2.0.8-1
2021-08-24 16:59:10 upgrade proxmox-backup-server:amd64 2.0.8-1 2.0.9-2
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.8.gz
2021-09-28 13:10:45 upgrade proxmox-backup-server:amd64 2.0.9-2 2.0.10-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.7.gz
2021-10-08 10:58:50 upgrade proxmox-backup-server:amd64 2.0.10-1 2.0.11-1
2021-10-21 19:03:51 upgrade proxmox-backup-server:amd64 2.0.11-1 2.0.12-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.6.gz
2021-11-09 20:08:48 upgrade proxmox-backup-server:amd64 2.0.12-1 2.0.13-1
2021-11-16 13:16:30 upgrade proxmox-backup-server:amd64 2.0.13-1 2.0.14-1
2021-11-23 16:17:23 upgrade proxmox-backup-server:amd64 2.0.14-1 2.1.2-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.5.gz
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.4.gz
2022-01-18 19:42:07 upgrade proxmox-backup-server:amd64 2.1.2-1 2.1.3-1
2022-01-25 20:14:34 upgrade proxmox-backup-server:amd64 2.1.3-1 2.1.4-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.3.gz
2022-02-06 13:02:45 upgrade proxmox-backup-server:amd64 2.1.4-1 2.1.5-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.2.gz
root@proxpbs:/var/log# grep "upgrade proxmox-backup-server" /var/log/dpkg.log.1
2022-04-24 13:16:10 upgrade proxmox-backup-server:amd64 2.1.5-1 2.1.6-1
root@proxpbs:/var/log# grep "upgrade proxmox-backup-server" /var/log/dpkg.log
2022-05-14 08:57:14 upgrade proxmox-backup-server:amd64 2.1.6-1 2.1.8-1
2022-05-20 22:33:50 upgrade proxmox-backup-server:amd64 2.1.8-1 2.2.1-1

 

[dcsapak]

ok, at the end of the garbage collection, we write that '.gc-status' file out (and thats the place where we read it from again). could you try to run another gc to see if that file exists afterwards ? maybe it's just a permission thing since you mount it via nfs?
(is the 'backup' user allowed to create files there?)

 

[JSEHV]

No, unfortunately after a GC task the file is not created.

Yes, the 'backup' user is allowed to write via nfs to the nas/san.
The 'backup' user is the 1001 user as seen in the posts before.



It looks like if the file is not present, PBS won't create it.
Even if I create the .gc-status file and fill it with the following content:

"disk-bytes":566582422000,"disk-chunks":229409,"index-data-bytes":6803777950000,"index-file-count":1311311,"pending-bytes":16287589700,"pending-chunks":7847,"removed-bad":0,"removed-bytes":0,"removed-chunks":0,"still-bad":0,"upid":null

the .gc-status file content is not updated after a GC task!

The content of the self created .gc-status file seems to be read correctly by PBS, as on the datastore it shows now a Deduplication Factor of 12.01
Also no errors in the syslog are shown (which it does when the string in the file is not the expected one).

PBS should create .gc-status if it is not present and update the .gc-status file content after each GC task.

(Remark: If the 'backup' user would not be able to write to the nas/san via nfs, also all backup folders and subfolders in the vm folder would not be created.)

 

[dcsapak]

PBS should create .gc-status if it is not present and update the .gc-status file content after each GC task.

yes and it normally does, but something is preventing it from doing so

i just noticed that the 'ls -la' shows '1001' instead of the 'backup' user, so maybe it's just an owner/permission problem after all?
here on a freshly installed pbs from iso the backup user has the uid/gid 34 not 1001
how did you setup your pbs ?