Deduplication not correctly shown

JSEHV

Member
Nov 11, 2020
20
9
8
52
System running: PBS 2.2-1 / Linux proxpbs 5.15.35-1-pve #1 SMP PVE 5.15.35-3

Deduplication not correctly shown (storage in PBS is supplied via NFS).

After the garbage collect run has done it's work, it shows a deduplication factor of 11.61

Screen Shot 2022-05-23 at 22.22.09.png

But when you look the summary of the datastore, it shows incorrectly still 1.00

Screen Shot 2022-05-23 at 22.22.37.png

Also in the stats of the datastore itself, it doesn't show the latest garbage collect statistics, but still 1.00

Screen Shot 2022-05-23 at 22.27.59.png
 
can you post the output of
Code:
proxmox-backup-debug api get /status/datastore-usage --output-format json
?
 
I've attached the json output to this reply in a plain text file.
 

Attachments

  • proxmox-backup-debug-jsehv.txt
    28 KB · Views: 9
ok the data is there, but it's empty/ all zeros...

can you post the content of the file .gc-status in the datastore?
for example if you datastore path is /mnt/datastore/foo:
Code:
cat /mnt/datastore/foo/.gc-status
 
  • Like
Reactions: Scynk159
There seem to be no .gc-status file in the datastore.
Only .lock and .chunks

I haven't manually adjusted anything, just installed PBS and upgraded over time.

Results from the prompt

Bash:
root@proxpbs:/# cd mnt/backup-pbs/
root@proxpbs:/mnt/backup-pbs# ls -la
total 8279
drwxr-xr-x     5 1001 1001     6 May 23 21:33 .
drwxr-xr-x     4 root root  4096 Jul 14  2021 ..
drwxr-x--- 65538 1001 1001 65538 Jul 14  2021 .chunks
-rw-r--r--     1 1001 1001     0 Jul 14  2021 .lock
drwxr-xr-x     2 1001 1001     2 May 21 18:12 ns
drwxr-xr-x    18 1001 1001    18 May 22 13:16 vm
root@proxpbs:/mnt/backup-pbs# cd ns
root@proxpbs:/mnt/backup-pbs/ns# ls -la
total 1
drwxr-xr-x 2 1001 1001 2 May 21 18:12 .
drwxr-xr-x 5 1001 1001 6 May 23 21:33 ..
root@proxpbs:/mnt/backup-pbs/ns# cd ..
root@proxpbs:/mnt/backup-pbs# cd vm
root@proxpbs:/mnt/backup-pbs/vm# ls -la
total 145
drwxr-xr-x 18 1001 1001 18 May 22 13:16 .
drwxr-xr-x  5 1001 1001  6 May 23 21:33 ..
drwxr-xr-x 13 1001 1001 14 May 25 03:00 111
drwxr-xr-x 13 1001 1001 14 May 25 03:03 112
drwxr-xr-x 13 1001 1001 14 May 25 03:06 113
drwxr-xr-x 13 1001 1001 14 May 25 03:08 114
drwxr-xr-x 13 1001 1001 14 May 25 03:00 201
drwxr-xr-x 13 1001 1001 14 May 25 03:01 202
drwxr-xr-x 13 1001 1001 14 May 25 03:02 203
drwxr-xr-x 13 1001 1001 14 May 25 03:02 204
drwxr-xr-x 13 1001 1001 14 May 25 03:02 205
drwxr-xr-x 13 1001 1001 14 May 25 03:03 208
drwxr-xr-x 13 1001 1001 14 May 25 03:03 209
drwxr-xr-x 13 1001 1001 14 May 25 03:00 321
drwxr-xr-x 13 1001 1001 14 May 25 03:00 322
drwxr-xr-x 13 1001 1001 14 May 25 03:04 900
drwxr-xr-x 13 1001 1001 14 May 25 03:04 901
drwxr-xr-x 13 1001 1001 14 May 25 03:04 902
root@proxpbs:/mnt/backup-pbs/vm#

Upgrade path after initial installation

Bash:
root@proxpbs:/var/log# zgrep "install proxmox-backup-server" /var/log/dpkg.log.10.gz
2021-07-14 12:47:18 install proxmox-backup-server:amd64 <none> 2.0.4-1

root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.10.gz
2021-07-22 08:48:22 upgrade proxmox-backup-server:amd64 2.0.4-1 2.0.5-2
2021-07-26 12:20:33 upgrade proxmox-backup-server:amd64 2.0.5-2 2.0.7-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.9.gz
2021-08-09 11:33:35 upgrade proxmox-backup-server:amd64 2.0.7-1 2.0.8-1
2021-08-24 16:59:10 upgrade proxmox-backup-server:amd64 2.0.8-1 2.0.9-2
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.8.gz
2021-09-28 13:10:45 upgrade proxmox-backup-server:amd64 2.0.9-2 2.0.10-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.7.gz
2021-10-08 10:58:50 upgrade proxmox-backup-server:amd64 2.0.10-1 2.0.11-1
2021-10-21 19:03:51 upgrade proxmox-backup-server:amd64 2.0.11-1 2.0.12-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.6.gz
2021-11-09 20:08:48 upgrade proxmox-backup-server:amd64 2.0.12-1 2.0.13-1
2021-11-16 13:16:30 upgrade proxmox-backup-server:amd64 2.0.13-1 2.0.14-1
2021-11-23 16:17:23 upgrade proxmox-backup-server:amd64 2.0.14-1 2.1.2-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.5.gz
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.4.gz
2022-01-18 19:42:07 upgrade proxmox-backup-server:amd64 2.1.2-1 2.1.3-1
2022-01-25 20:14:34 upgrade proxmox-backup-server:amd64 2.1.3-1 2.1.4-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.3.gz
2022-02-06 13:02:45 upgrade proxmox-backup-server:amd64 2.1.4-1 2.1.5-1
root@proxpbs:/var/log# zgrep "upgrade proxmox-backup-server" /var/log/dpkg.log.2.gz
root@proxpbs:/var/log# grep "upgrade proxmox-backup-server" /var/log/dpkg.log.1
2022-04-24 13:16:10 upgrade proxmox-backup-server:amd64 2.1.5-1 2.1.6-1
root@proxpbs:/var/log# grep "upgrade proxmox-backup-server" /var/log/dpkg.log
2022-05-14 08:57:14 upgrade proxmox-backup-server:amd64 2.1.6-1 2.1.8-1
2022-05-20 22:33:50 upgrade proxmox-backup-server:amd64 2.1.8-1 2.2.1-1
 
Last edited:
ok, at the end of the garbage collection, we write that '.gc-status' file out (and thats the place where we read it from again). could you try to run another gc to see if that file exists afterwards ? maybe it's just a permission thing since you mount it via nfs?
(is the 'backup' user allowed to create files there?)
 
No, unfortunately after a GC task the file is not created.

Yes, the 'backup' user is allowed to write via nfs to the nas/san.
The 'backup' user is the 1001 user as seen in the posts before.

Screen Shot 2022-06-04 at 10.22.07.png

It looks like if the file is not present, PBS won't create it.
Even if I create the .gc-status file and fill it with the following content:

Code:
"disk-bytes":566582422000,"disk-chunks":229409,"index-data-bytes":6803777950000,"index-file-count":1311311,"pending-bytes":16287589700,"pending-chunks":7847,"removed-bad":0,"removed-bytes":0,"removed-chunks":0,"still-bad":0,"upid":null

the .gc-status file content is not updated after a GC task!

The content of the self created .gc-status file seems to be read correctly by PBS, as on the datastore it shows now a Deduplication Factor of 12.01
Also no errors in the syslog are shown (which it does when the string in the file is not the expected one).

PBS should create .gc-status if it is not present and update the .gc-status file content after each GC task.

(Remark: If the 'backup' user would not be able to write to the nas/san via nfs, also all backup folders and subfolders in the vm folder would not be created.)
 
Last edited:
PBS should create .gc-status if it is not present and update the .gc-status file content after each GC task.
yes and it normally does, but something is preventing it from doing so

i just noticed that the 'ls -la' shows '1001' instead of the 'backup' user, so maybe it's just an owner/permission problem after all?
here on a freshly installed pbs from iso the backup user has the uid/gid 34 not 1001
how did you setup your pbs ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!