[SOLVED] Backup looks empty but works

[sabiwww]

I am trying out PVE and PBS. Installed it on a test PC using two SSDs. The setup:
- PVE installed on a ZFS RAID1 2x2TB SSD
- PVE and PBS on the same OS
- Added PBS as a remote on PVE
- The PBS storage points to the path /backup
- Registered the backup job on PVE targeting PBS

When I run a backup it says it transferred some data, like:

()
INFO: trying to get global lock - waiting...
INFO: got global lock
INFO: starting new backup job: vzdump --storage BackupServer --mode snapshot --quiet 1 --notes-template '{{guestname}}' --all 1 --mailnotification failure
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2022-07-09 21:00:04
INFO: status = running
INFO: VM Name: windows
INFO: include disk 'scsi0' 'local-zfs:vm-101-disk-0' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2022-07-09T19:00:04Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'a8e0b343-a835-46db-83d5-3435034b3acd'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (5.0 GiB of 64.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 5.0 GiB dirty of 64.0 GiB total
INFO:  10% (544.0 MiB of 5.0 GiB) in 3s, read: 181.3 MiB/s, write: 181.3 MiB/s
INFO:  19% (1012.0 MiB of 5.0 GiB) in 6s, read: 156.0 MiB/s, write: 156.0 MiB/s
INFO:  28% (1.4 GiB of 5.0 GiB) in 9s, read: 150.7 MiB/s, write: 150.7 MiB/s
INFO:  37% (1.9 GiB of 5.0 GiB) in 12s, read: 146.7 MiB/s, write: 146.7 MiB/s
INFO:  46% (2.3 GiB of 5.0 GiB) in 15s, read: 160.0 MiB/s, write: 160.0 MiB/s
INFO:  53% (2.7 GiB of 5.0 GiB) in 18s, read: 112.0 MiB/s, write: 112.0 MiB/s
INFO:  65% (3.3 GiB of 5.0 GiB) in 21s, read: 210.7 MiB/s, write: 210.7 MiB/s
INFO:  76% (3.8 GiB of 5.0 GiB) in 24s, read: 182.7 MiB/s, write: 182.7 MiB/s
INFO:  86% (4.3 GiB of 5.0 GiB) in 27s, read: 174.7 MiB/s, write: 173.3 MiB/s
INFO:  96% (4.8 GiB of 5.0 GiB) in 30s, read: 169.3 MiB/s, write: 169.3 MiB/s
INFO: 100% (5.0 GiB of 5.0 GiB) in 32s, read: 98.0 MiB/s, write: 98.0 MiB/s
INFO: backup was done incrementally, reused 59.00 GiB (92%)
INFO: transferred 5.01 GiB in 32 seconds (160.2 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 101 (00:00:33)
INFO: Backup finished at 2022-07-09 21:00:37
INFO: Backup job finished successfully
TASK OK

However running `ls -lh` on the backup path gives me this:

root@pve:~# ls -lh /backup/vm/101/2022-07-09T19\:00\:04Z/
total 206K
-rw-r--r-- 1 backup backup  744 Jul  9 21:00 client.log.blob
-rw-r--r-- 1 backup backup 516K Jul  9 21:00 drive-scsi0.img.fidx
-rw-r--r-- 1 backup backup  416 Jul  9 21:00 index.json.blob
-rw-r--r-- 1 backup backup  444 Jul  9 21:00 qemu-server.conf.blob

How can the backup size be 516K? It said it transferred 5.01GB. It is the same for the all of the backups (even the first one, it is only 546K). I downloaded the whole /backup folder using SFTP and it only takes ~2MB.
If I delete some files and restore the vm to the last backup, it works. The deleted files are restored. So it must work, just not how I expect it.
Where are the backups stored? Shouldn't it be on the path PBS uses?

 

[Dunuin]

PBS chops everything into small chunk files of max 4MB in size that get deduplicated in the hidden /backups/.chunks" folder.

 

[sabiwww]

PBS chops everything into small chunk files of max 4MB in size that get deduplicated in the hidden /backups/.chunks" folder.

Oh, I guess you are right. I missed that. Thanks.

du says it takes 18GBs and the UI says usage is 30.86GB. What makes the difference?

root@pve:/# du -hs /backup/.chunks
18G     /backup/.chunks
root@pve:/# du -hs /backup
18G     /backup

 

[Dunuin]

Oh, I guess you are right. I missed that. Thanks.

du says it takes 18GBs and the UI says usage is 30.86GB. What makes the difference?

root@pve:/# du -hs /backup/.chunks
18G     /backup/.chunks
root@pve:/# du -hs /backup
18G     /backup

Which UI? Chunks are deduplicated so there is no way to see how big a single backup is. You can only see the size of your datastore which is the size of all backups you created so far.

 

[sabiwww]

Which UI? Chunks are deduplicated so there is no way to see how big a single backup is. You can only see the size of your datastore which is the size of all backups you created so far.

Yes. That one I meant.

And running du says:

root@pve:/# du -hs /backup
18G     /backup

 

[Dunuin]

Not sure. Maybe someone of the staff can explain how PBS is calculation the datastore size after the weekend.

 

[sabiwww]

Just a guess, but maybe the usage displays the overall storage space usage, not the actual datastore's usage.

 

[dcsapak]

Just a guess, but maybe the usage displays the overall storage space usage, not the actual datastore's usage.

it's this. actually getting the usage from the chunks may work on small datastores, but when it's getting bigger, the usage gathering doesn't scale well, so we simply stat the filesystem for usage
it does not interfere with functionality though, it's just the display. so if you want this display to be accurately reflecting the actual datastore usage, you should put it on it's own filesystem