What-if I delete index folders?

[Matteo Calorio]

Hello,

I see in my datastore I have a structure like that: vm -> VMid -> folders with timestamp name:

2022-04-02T09:33:01Z/
2022-04-09T01:18:16Z/
2022-04-12T04:02:06Z/
2022-04-13T01:57:38Z/
2022-04-14T02:48:12Z/
2022-04-15T02:47:03Z/
2022-04-16T02:41:47Z/
2022-04-19T04:47:29Z/
2022-04-20T02:45:08Z/
2022-04-22T04:19:43Z/
2022-04-23T04:55:08Z/
2022-04-26T03:48:27Z/

I would like to better undestand this behaviour: what if run "find ./ -type d -mtime +10 -delete"? Will I keep all "last 10 days" backups?

And will next garbage collection simply remove chunks not referenced anymore by deleted indexes?

Thanks,
M.

 

[dcsapak]

I would like to better undestand this behaviour: what if run "find ./ -type d -mtime +10 -delete"? Will I keep all "last 10 days" backups?

no, because the mtime might not correspon with the actual backup time (e.g. if you change the comment, or if you synced such a backup)

And will next garbage collection simply remove chunks not referenced anymore by deleted indexes?

yes, the garbage collect will remove all chunks that are not referenced anymore (after 24h+5min)

 

[Matteo Calorio]

no, because the mtime might not correspon with the actual backup time (e.g. if you change the comment, or if you synced such a backup)


yes, the garbage collect will remove all chunks that are not referenced anymore (after 24h+5min)

Thanks for you reply, but if I had just for example:

2022-04-23T04:55:08Z/
2022-04-26T03:48:27Z/

and I delete:

2022-04-23T04:55:08Z/

will I have a consistent (full) backup for the 26th of April?

I mean: are in each folder all necessary infos to restore a backup (or full for 23rd of Arpril and just incremental for 26th)?

 

[fabian]

all snapshots are created equal and are fully self-contained/independent, and the snapshot dirs themselves only contain metadata/indices. the actual data is in the (hidden) .chunks directory and will not be modified by pruning (whether through the API, or behind PBS' back like you deleting snapshot dirs manually ).

 

[Matteo Calorio]

Thank you very much! The reason I asked is that I suspect there is one or more corrupted index after a physical problem on the SAN we use.

The effect is that the garbage collection stucks at 9% for several days and never finishes (and disk space usage is 96%), so I would like to delete all index directories till the day we repaired the SAN to accelerate GC phase1, but I was worried about the following backups.

By the way... is it exactly the same to click on the trash bin in the web interface (permanently forget snapshot) for each snapshot done in 2021 and to run something like (in the datastore root):

find ./host ./vm -type d -name "2021*" -delete

 

[Matteo Calorio]

Hello,

in the end the solution was to remove all but one backup for each VM or host from the web interface to lighten the GC and let the GC go on. After 9 days it completed phase 1 and started to free up space with phase 2.

For the future: consider the significant inertia of the GC and set an appropriate retention.

M.

 

[Wolff]

Out of curiosity, what kind of Storage / CPU cycles do you have dedicated to your PBS ?

Ours runs on 2x Intel M.2's in zfsRaid1 and 8x 14 TB WD in zfsRaidZ1 and backs up about 20 TB of VM Data. Prune and GC does not take longer than 4 hours ( that is the Window we have allotted for Prune, GC and Remote Sync)

 

[Matteo Calorio]

Ours run on a test system right now, that is a VM with 8 cores (48 x Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz (2 Sockets)), 128GB of RAM, 10Gbps network. The storage is a 100TB space on a SAN via NFS on 10Gbps network. The SAN is a QNAP with Xeon E-2236 CPU (6 cores by 12 threads)and 16 6Gbps SATA drives in RAIDZ1. (I know it's not the best solution)

 

[Wolff]

Your bottleneck should be the single 10G line (1.25 GByte per second) that your SAN serves the VM/Host with.

For reference sake:
We use a PBS installed side by side on a Proxmox host (for quorum; hosts only lightweight test-vms)

• CPU: AMD EPYC 7313 (16 Cores a 3.0 Ghz)
• Proxmox <-> PBS connectivity is a single 100G line (just shy of 12 GByte/s transfer)
• PBS to remote Target is a 40G Network (handles just shy of 5 GByte/s) connected to a Synology NAS running BTRFS

 

[Matteo Calorio]

Hmm... I checked and we do not exceed 10% of the line capacity during normal operations (backup, verify, garbage collection).

 

[Q-wulf]

Have you checked CPU-utilisation using something like e.g. htop or nmon. Also benchmarked the throughput/IOps of your PBS-VM??

 

[Matteo Calorio]

Yes, thanks, CPU is low (< 7%) and the throughput, if for example I move a file via rsync from PBS to SAN or vice-versa, is 10 times more than normal activity on PBS