Size on special device for metadata cache

[PelleH]

Hi,
I will set up a Proxmox backup server to backup our Proxmox clusters. There are total 9 nodes (6 in one and 3 nodes in one). There are approximately 40 VM and they are using 10TB disk space.
I will set up maybe 6 SAS 18TB spinning disks for this and now my question:

How big should the special device mirror be? Se https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements
Will the only be used for small amount of data or will the metadata grow?
Are 256GB good or 1TB?

I will probably using 2 NVMe disks for this special device mirror.

/PelleH

 

[fabian]

you haven't said how your pool will be structured (3x2 mirror? 2x3 mirrors? raidz-1? raidz-2? raidz-3?). I am seeing about 20G special device usage for 1TB data usage (mixed PBS and VM storage), but it depends a lot on what kind of data you are backing up and how compressible it is.

you can probably get a good estimate by doing an initial backup run of a few of your VMs without a special device, then checking how much metadata those backups take up (e.g. using zdb), and extrapolating from there. note that you need to clear your datastore and redo those backups (or otherwise rewrite all the chunk files) after adding the special vdevs in order to see it in effect.

 

[PelleH]

The 6 disk will probably be a Raidz-2. All backups will be VM storage. There are a mix of Windows and Linux VM.

 

[Cookiefamily]

After using ZFS in a lot of scenarios we developed a rule of thumb of a 0.02 ratio for the special device. So per 1TB if storage 20gb of special device. This alignes with what @fabian uses + some safety headroom.

 

[PelleH]

Thanks. That will be big disks for special device. If we have around 60TB it will be 1,8TB.

 

[Cookiefamily]

Whoops I did mess up when looking in our wiki, it was 0.02. you will use less when you have less files on the system, but since PBS tries to hit 4MiB files you will have quite the amount of metadata.

 

[PelleH]

Ok, thanks for that. I did order two 1,6TB NVMe and with 0,02 it will complete the task.
I will try with those disks and see how much space metadata takes up after a few weeks of backup.

 

[oversite]

Ok, thanks for that. I did order two 1,6TB NVMe and with 0,02 it will complete the task.
I will try with those disks and see how much space metadata takes up after a few weeks of backup.

Our special device vdev is using 3,3GB for 1TB backup data. Mirrored vdevs, have been running daily backups for a month.