PBS on RAIDZ2+HDD, no special file

[cyruspy]

Hello,

I know it has been mentioned several times a special device backed by ssd is mandatory when using HDDs for data to have a bearable solution.

If I don't have SSD but RAM is available (128GB), could ARC save me here?

 

[Johannes S]

It helps a bit but not much compared to a Special device on a ssd mirror. You could do some Benchmark to get data instead of guessing.

 

[UdoB]

Without an SD all metadata is stored on the HDDs. To read it, the head has to move physically. That takes some milliseconds.

As all data (including meta-data) is equally distributed on the whole device this repeats often. Unfortunately I have no idea at how many different locations meta data is actually stored, but I assume many. With .chunks containing a 6-digit number of files (or more!) that is.

All of these physical movements are completely absorbed by an independent Special Device.

Regarding Ram usage for ARC: it may possibly help to store only metadata in Ram. See man zfsprops : primarycache=all|none|metadata. Not tested by me, I have Special Devices

I am unsure what the state of "persistent ARC" is nowadays. If ARC starts empty then all metadata has to be read at least once...

 

[VictorSTS]

I know it has been mentioned several times a special device backed by ssd is mandatory when using HDDs for data to have a bearable solution.

Yes, but not every single time. Depends essentially on the size of the datastore, how much work PBS has to do (i.e. backup once a day vs once an hour, many replicas, many verify, etc), and how much time you have to run GC, which is the task that really requires an special device. I.e. I don't use special device in small-ish PBS with basic loads, say up to 6TB-backup once a day -> Im fine if GC takes some hours.

If I don't have SSD but RAM is available (128GB), could ARC save me here?

Not at all: ARC is a read cache. Even if you had terabytes of RAM just for ARC:

• Writes to metadata done on every GC to update access time for every chunk will have to be written to disk no matter what.
• It will be lost on server reboot/power failure/crash.

A big ARC somewhat helps on verify or sync tasks, but usually the datastore is way bigger that any reasonable amount of RAM you set for ARC, so a very big ARC benefit is quite limited.

Regarding Ram usage for ARC: it may possibly help to store only metadata in Ram. See man zfsprops : primarycache=all|none|metadata.

I did test this some years ago on different hardware configurations and primarycache=metadata did reduce GC times by ~4 but increased every other task by at least a x3 factor (syncs took a big hit around x6!). Wouldn't recommend it unless you can test it with your hardware and exact workload (i.e. you really need to shorten GC and don't care about everything else being slower).


PS: ARC is misbehaving badly on kernel 7.0 [1]

[1] https://forum.proxmox.com/threads/kernel-7-0-performance-issue-with-zfs-pools.183872/post-866334

 

[cyruspy]

Thanks, will try to split the boot hardware RAID and use a ZFS mirrored partition as special device.