PBS - Total backup sizes just keeps going up and up

[harmonyp]

Prune & GC job running daily and a verify job running daily

Second picture is the total disk space of PBS, it just keeps going up and up even though there are not more backups being added (maxiumum of 3)

Last part of the image is my ZFS pool size hosting the virtual machines, it's not going up

Why is this happening? it's like the old backups are not being removed.

 

[Dunuin]

Check if there are any snapshots more misconfigured discard preventing the storage from freeing up space of deleted chunks.
In case your PBS datastore is backed by ZFS the output of zpool list -v should be useful. USEDREFRESERV and USEDSNAP should be low.

 

[harmonyp]

Check if there are any snapshots more misconfigured discard preventing the storage from freeing up space of deleted chunks.
In case your PBS datastore is backed by ZFS the output of zpool list -v should be useful. USEDREFRESERV and USEDSNAP should be low.

root@storage:~# zpool list -v
NAME                            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfs                             104T  42.3T  61.5T        -         -     0%    40%  1.00x    ONLINE  -
  raidz2-0                      102T  42.1T  59.7T        -         -     0%  41.4%      -    ONLINE
    wwn-0x5000cca28f6d86a0     12.7T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca28f6cfd04     12.7T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca28f6c7a3c     12.7T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca2ad24cbb8     12.7T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca2ad2e3384     12.7T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca28f4c9bd8     12.7T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca28f4292d0     12.7T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca2905e8898     12.7T      -      -        -         -      -      -      -    ONLINE
special                            -      -      -        -         -      -      -      -  -
  mirror-1                     1.86T   118G  1.74T        -         -     8%  6.20%      -    ONLINE
    nvme-eui.002538b621b5ed26  1.86T      -      -        -         -      -      -      -    ONLINE
    nvme-eui.002538b621b5ee35  1.86T      -      -        -         -      -      -      -    ONLINE
root@storage:~# zfs list -t snapshot
NAME                                  USED  AVAIL     REFER  MOUNTPOINT
zfs/VMDATA/base-100-disk-0@__base__     8K      -     1.65G  -
root@storage:~#

 

[Dunuin]

Ah sorry, it's zfs list -o space for USEDSNAP and USEDREFRESERV.

 

[harmonyp]

Ah sorry, it's zfs list -o space for USEDSNAP and USEDREFRESERV.

root@storage:~# zfs list -o space
NAME                         AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zfs                          42.2T  30.1T        0B    104K             0B      30.1T
zfs/PBS                      42.2T  26.0T        0B    104K             0B      26.0T
zfs/PBS/Havoc               42.2T  24.1T        0B   24.1T             0B         0B
zfs/PBS/Claire                42.2T  1.89T        0B   1.89T             0B         0B
zfs/VMDATA                   42.2T  4.05T        0B     96K             0B      4.05T
zfs/VMDATA/base-100-disk-0   42.2T  1.65G        8K   1.65G             0B         0B
zfs/VMDATA/vm-100-cloudinit  42.2T    56K        0B     56K             0B         0B
zfs/VMDATA/vm-999-cloudinit  42.2T  81.1K        0B   81.1K             0B         0B
zfs/VMDATA/vm-999-disk-0     42.2T  4.05T        0B   4.05T             0B         0B
root@storage:~#

 

[fabian]

the GC log contains information about chunks it removes, so you can check there if it does what you expect it to do. it's possible that your VMs are not discarding no longer needed data properly, and you are backing that up and thus accumulating more and more "backup data".

 

[harmonyp]

the GC log contains information about chunks it removes, so you can check there if it does what you expect it to do. it's possible that your VMs are not discarding no longer needed data properly, and you are backing that up and thus accumulating more and more "backup data".

Is logging enabled by default? I can't find anything in /var/log/proxmox-backup/tasks about GC

 

[fabian]

yes. you can browse the tasks (and their logs) on the GUI as well:

Administraiton -> Tasks -> Task Type: garbage_collection and then double click on any entry in the list. the end of a GC log should look somewhat like this:

...
2023-04-26T09:09:12+02:00: processed 99% (31184 chunks)
2023-04-26T09:09:12+02:00: Removed garbage: 5.035 GiB
2023-04-26T09:09:12+02:00: Removed chunks: 2528
2023-04-26T09:09:12+02:00: Pending removals: 1.341 GiB (in 1123 chunks)
2023-04-26T09:09:12+02:00: Removed bad chunks: 2
2023-04-26T09:09:12+02:00: Original data usage: 1.228 TiB
2023-04-26T09:09:12+02:00: On-Disk usage: 54.903 GiB (4.37%)
2023-04-26T09:09:12+02:00: On-Disk chunks: 27837
2023-04-26T09:09:12+02:00: Deduplication factor: 22.90
2023-04-26T09:09:12+02:00: Average chunk size: 2.02 MiB
2023-04-26T09:09:12+02:00: TASK OK

"Removed" means things removed in this run, Pending means things which will likely be removed in the near future, but safeguards prevent removal at the moment (usually this means chunks which are not referenced, but their atime is after the cutoff, so PBS is not 100% sure whether they might have been added by an ongoing backup task).