[SOLVED] Storage space requirements increased

[andreasc]

Hi Forum,

since 03/11 I have noticed that the storage space requirement has increased significantly. From round about 700GB to now 2,2TB with a rate of 200GB per day.
I backup only LXC Container with alot of static files (photos, music, docs..) and a low changing rate.



Package Versions of the backup server:
proxmox-backup: 1.0-4 (running kernel: 5.4.103-1-pv
proxmox-backup-server: 1.0.9-1 (running version: 1.0.9)
pve-kernel-5.4: 6.3-7
pve-kernel-helper: 6.3-7
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.101-1-pve: 5.4.101-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
proxmox-backup-docs: 1.0.9-1
proxmox-backup-client: 1.0.9-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-6
pve-xtermjs: 4.7.0-3
smartmontools: 7.2-pve2
zfsutils-linux: 2.0.3-pve2

Package Versions of the Proxmox VE server:
proxmox-ve: 6.3-1 (running kernel: 5.4.103-1-pve) pve-manager: 6.3-6 (running version: 6.3-6/2184247e) pve-kernel-5.4: 6.3-7 pve-kernel-helper: 6.3-7 pve-kernel-5.4.103-1-pve: 5.4.103-1 pve-kernel-5.4.101-1-pve: 5.4.101-1 ceph-fuse: 12.2.11+dfsg1-2.1+b1 corosync: 3.1.0-pve1 criu: 3.11-3 glusterfs-client: 5.5-3 ifupdown2: 3.0.0-1+pve3 ksm-control-daemon: 1.3-1 libjs-extjs: 6.0.1-10 libknet1: 1.20-pve1 libproxmox-acme-perl: 1.0.7 libproxmox-backup-qemu0: 1.0.3-1 libpve-access-control: 6.1-3 libpve-apiclient-perl: 3.1-3 libpve-common-perl: 6.3-5 libpve-guest-common-perl: 3.1-5 libpve-http-server-perl: 3.1-1 libpve-storage-perl: 6.3-7 libqb0: 1.0.5-1 libspice-server1: 0.14.2-4~pve6+1 lvm2: 2.03.02-pve4 lxc-pve: 4.0.6-2 lxcfs: 4.0.6-pve1 novnc-pve: 1.1.0-1 proxmox-backup-client: 1.0.10-1 proxmox-mini-journalreader: 1.1-1 proxmox-widget-toolkit: 2.4-6 pve-cluster: 6.2-1 pve-container: 3.3-4 pve-docs: 6.3-1 pve-edk2-firmware: 2.20200531-1 pve-firewall: 4.1-3 pve-firmware: 3.2-2 pve-ha-manager: 3.1-1 pve-i18n: 2.2-2 pve-qemu-kvm: 5.2.0-3 pve-xtermjs: 4.7.0-3 qemu-server: 6.3-8 smartmontools: 7.2-pve2 spiceterm: 3.1-1 vncterm: 1.6-2 zfsutils-linux: 2.0.3-pve2

Any idea? What kind of information I should provide?
Can this be related to the ZFS changes?
oot@pbs01:~# zpool upgrade rpool
This system supports ZFS pool feature flags.

Enabled the following features on 'rpool':
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress

Best regards
Andreas

 

[fabian]

possibly this issue: https://forum.proxmox.com/threads/p...ackup-time-after-some-days.85703/#post-376587

can you verify with /var/log/apt/history.log on the PVE side whether the start of the slop co-incides with an upgrade, and if so, which proxmox-backup-client versions were involved?

 

[andreasc]

Hi Fabian,

root@pve1:~# proxmox-backup-client version
client version: 1.0.10

Attached you will find the apt history from pve and pbs.

Best regards
Andreas

 

[fabian]

thanks. seems like 1.0.9-1 and 1.0.10-1 are affected, and the soon-to-be-released 1.0.11-1 hopefully fixes it for you as well. once it is out and you made a few backups with it, you might want to remove the snapshots to regain the space.

 

[andreasc]

Hi Fabian,

thank you for your fast replies. I'll will give you a feedback after the update and some backups.

Best regards
Andreas

 

[andreasc]

Hi Fabian

a last question:
I have delete all snapshots since the 03/11. The garbe collection and the prune job are finished successfully. How can I cleanup the pool?

root@pbs1:~# zfs list -t filesystem -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
rpool 77.7M 2.55T 0B 140K 0B 2.55T
rpool/ROOT 77.7M 2.55T 0B 140K 0B 2.55T
rpool/ROOT/pbs-1 77.7M 2.55T 0B 2.55T 0B 0B

root@pbs1:~# zfs list -r -t snapshot -o name,creation rpool/ROOT/pbs-1
no datasets available
root@pbs1:~# zfs list -r -t snapshot -o name,creation rpool
no datasets available
root@pbs1:~# zfs list -r -t snapshot -o name,creation rpool/ROOT
no datasets available


Status email:
Datastore: pve1
Task ID: UPIDbs1:00000A99:00001B50:00000002:6051ECD9:garbage_collectionve1:root@pam:
Index file count: 108

Removed garbage: 0 B
Removed chunks: 0
Removed bad chunks: 0

Leftover bad chunks: 0
Pending removals: 1.94 TiB (in 480684 chunks)

Original Data usage: 3.82 TiB
On-Disk usage: 613.13 GiB (15.68%)
On-Disk chunks: 181351

Deduplication Factor: 6.38

Best regards
Andreas

 

[cryonie]

Hello,
Maybe i'm wrong but i"ve read somewhere that garbage collection delete unused chunks ONLY after they are not used for 24+h.
So if you deleted your backups today, wayt 24h and then do the garbage collection and ... tada plenty of space

 

[fabian]

yes. the amount listed as 'pending removals' is still so new that we can't tell for sure whether it is still used - see https://pbs.proxmox.com/docs/backup-client.html#client-garbage-collection for details

 

[andreasc]

Thank you for the link and explenation. The garbe collection now run's, but stuck since 2021-03-18T23:58:12+01:00: marked 55% (60 of 108 index files).
I'll will wait...

Best reagards

Andreas

 

[andreasc]

So now it is ready, 1 day an 5 hours later the datastore is now clean. Thanks a lot for your fast answers and the hint with the documentation.

Best regards

Andreas

 

[andreasc]

Since the update yesterday to proxmox-backup-client 1.0.11-1 on the pve and the update of the pbs the storage usage is normal again.
Thanks for your support.