PBS on RAIDZ2+HDD, no special file
- Thread starter cyruspy
- Start date
Without an SD all metadata is stored on the HDDs. To read it, the head has to move physically. That takes some milliseconds.
As all data (including meta-data) is equally distributed on the whole device this repeats often. Unfortunately I have no idea at how many different locations meta data is actually stored, but I assume many. With .chunks containing a 6-digit number of files (or more!) that is.
All of these physical movements are completely absorbed by an independent Special Device.
Regarding Ram usage for ARC: it may possibly help to store only metadata in Ram. See
I am unsure what the state of "persistent ARC" is nowadays. If ARC starts empty then all metadata has to be read at least once...
As all data (including meta-data) is equally distributed on the whole device this repeats often. Unfortunately I have no idea at how many different locations meta data is actually stored, but I assume many. With .chunks containing a 6-digit number of files (or more!) that is.
All of these physical movements are completely absorbed by an independent Special Device.
Regarding Ram usage for ARC: it may possibly help to store only metadata in Ram. See
man zfsprops : primarycache=all|none|metadata. Not tested by me, I have Special Devices I am unsure what the state of "persistent ARC" is nowadays. If ARC starts empty then all metadata has to be read at least once...
Yes, but not every single time. Depends essentially on the size of the datastore, how much work PBS has to do (i.e. backup once a day vs once an hour, many replicas, many verify, etc), and how much time you have to run GC, which is the task that really requires an special device. I.e. I don't use special device in small-ish PBS with basic loads, say up to 6TB-backup once a day -> Im fine if GC takes some hours.
Not at all: ARC is a read cache. Even if you had terabytes of RAM just for ARC:
- Writes to metadata done on every GC to update access time for every chunk will have to be written to disk no matter what.
- It will be lost on server reboot/power failure/crash.
I did test this some years ago on different hardware configurations and
primarycache=metadata did reduce GC times by ~4 but increased every other task by at least a x3 factor (syncs took a big hit around x6!). Wouldn't recommend it unless you can test it with your hardware and exact workload (i.e. you really need to shorten GC and don't care about everything else being slower).PS: ARC is misbehaving badly on kernel 7.0 [1]
[1] https://forum.proxmox.com/threads/kernel-7-0-performance-issue-with-zfs-pools.183872/post-866334
I would put the RAID controller in JBOD / IT mode to convert it to an HBA, the use ZFS for both the OS and special device. Remember that ZFS isn't supported on RAID controllers/LUNs. When installing PVE/PBS, set the disk size to ~50G, should be enough for PBS, even after some time (PBS may need some space for logs). Then, create partitions on remaining space and use a mirror on them with special device. You can practice this setup on a VM.
This is only HDD, zero SSD. (As of #1)
Having a "separate" Special Device on the very same spindles makes no sense.
Actually we know not much at all. What actual drives are available? How many? How many (unused) connectors has the controller? It "IT-mode" achievable? Is buying two small SSDs an option? "Guessing" is always sub-optimal...
You right, somehow I understood that there are SSD for OS but not SSD to dedicate to SD, so imagined that OP wanted to devote some space on those SSD for SD.
I'm not aware of a bug that won't allow install, mind posting a link to some reference?
If your hardware RAID controller has a battery-backed RAM cache, you might have bearable performance with write-back enabled. Have you already tried some PBS tasks and are needing to improve performance? Or is this still in the building phase?
Of course the special device must be reliable once it is attached to the disks. So while running the SSDs via a RAID controller has some negatives, if you have monitoring and scrubs set up on the RAID controller, I wouldn't have major concerns with running ZFS on it.
Of course the special device must be reliable once it is attached to the disks. So while running the SSDs via a RAID controller has some negatives, if you have monitoring and scrubs set up on the RAID controller, I wouldn't have major concerns with running ZFS on it.
If I recall correctly it's entirely possible to use ZFS with HW-RAID if (and only if!) you don't build a pool but instead uses the virtual disc provided by the RAID-Controller in a single-device-pool. @Falk R. mentioned such a setup in the German forum and might point out possible caveats and If I remembered wrong. One obvious caveat would be, that you won't have ZFS autohealing (as long as you don't set the copies parameter to 2) but with PBS own verification and another PBS as offsite backup this doesn't need to be a problem imho.