Size on special device for metadata cache

[PelleH]

Hi,
I will set up a Proxmox backup server to backup our Proxmox clusters. There are total 9 nodes (6 in one and 3 nodes in one). There are approximately 40 VM and they are using 10TB disk space.
I will set up maybe 6 SAS 18TB spinning disks for this and now my question:

How big should the special device mirror be? Se https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements
Will the only be used for small amount of data or will the metadata grow?
Are 256GB good or 1TB?

I will probably using 2 NVMe disks for this special device mirror.

/PelleH

 

[fabian]

you haven't said how your pool will be structured (3x2 mirror? 2x3 mirrors? raidz-1? raidz-2? raidz-3?). I am seeing about 20G special device usage for 1TB data usage (mixed PBS and VM storage), but it depends a lot on what kind of data you are backing up and how compressible it is.

you can probably get a good estimate by doing an initial backup run of a few of your VMs without a special device, then checking how much metadata those backups take up (e.g. using zdb), and extrapolating from there. note that you need to clear your datastore and redo those backups (or otherwise rewrite all the chunk files) after adding the special vdevs in order to see it in effect.

 

[PelleH]

The 6 disk will probably be a Raidz-2. All backups will be VM storage. There are a mix of Windows and Linux VM.

 

[Cookiefamily]

After using ZFS in a lot of scenarios we developed a rule of thumb of a 0.02 ratio for the special device. So per 1TB if storage 20gb of special device. This alignes with what @fabian uses + some safety headroom.

 

[PelleH]

Thanks. That will be big disks for special device. If we have around 60TB it will be 1,8TB.

 

[Cookiefamily]

Whoops I did mess up when looking in our wiki, it was 0.02. you will use less when you have less files on the system, but since PBS tries to hit 4MiB files you will have quite the amount of metadata.

 

[PelleH]

Ok, thanks for that. I did order two 1,6TB NVMe and with 0,02 it will complete the task.
I will try with those disks and see how much space metadata takes up after a few weeks of backup.

 

[oversite]

Ok, thanks for that. I did order two 1,6TB NVMe and with 0,02 it will complete the task.
I will try with those disks and see how much space metadata takes up after a few weeks of backup.

Our special device vdev is using 3,3GB for 1TB backup data. Mirrored vdevs, have been running daily backups for a month.

 

[Fathi]

Hi,
Using commodity hardware for PBS, on which it's not possible to install more than one nvme ssd in case it supports nmv. Datacenter class devices either spinning discs or ssd also either U2 or nvme are not available on all countries. Datacenter class devices are sold and only available within bigger solutions (software + hardware) either on purchase or as spare parts. So, the only available solutions for me are commodity hardware. I try to gather information on the net about every piece of hardware I buy before buying it : TBW, MTBF, ...
Having experienced extremely slow vm restoration time (73 hours for 4TB), i have been advised in one of the posts to put a special device on my PBS.
Knowing that I usually use two PBS servers, one local for backups and one remote synchronizing from the first, or when using a single PBS, the backed-up vm being a backup of a nas that doesn't have a pbs client like truenas; what are the risks if i use a commodity sata ssd as a special device ? I could double the ssd size, so the controller will provision new cells when some of them fail. Does this mitigate the risk of a special device failure or are you talking about other causes of failure, not just those of the storage cells ?
TIA.
Fathi B.N.