[SOLVED] Benefit special device?

[cdn123]

Hi,

We’re running a setup with 2× 1TB NVMe SSDs and 10× 16TB HDDs. I’m currently undecided between using RAIDZ-2 or 5× mirrored vdevs for the HDDs.

The HDD storage hosts several virtual disks (vdisk), which are accessed via SMB from file server for other VMs. System disks are stored at nvme.

I also have a question regarding the NVMe drives:Would it be beneficial to allocate part of the NVMe storage as a special device in ZFS?I’m unsure whether VM metadata would be stored there as well. If not, I’m hesitant to sacrifice NVMe capacity without a clear advantage.

Thanks in advance for your help!

 

[UdoB]

Go for mirrored drives. A single RaidZ2 gives you the IOPS of a SINGLE drive; two five-drive RaidZ2 --> ... of two drives.

Five mirrors gives... the IOPS of five of them. OF course this is still slow!

Whatever you put into a ZFS pool the metadata will be stored on the Special Device. The effects I have noticed during the last few years made me to never setup a new pool with rotating rust without it


Disclaimer: of course it depends on the actual usage and use case, but more IOPS is often... very welcome.

Edit: if you go for RaidZ2 the SD should be a tripple mirror - the redundancy of "two devices are allowed to fail" should be identical.

 

[cdn123]

Yeah I thought about these options and decide to go with 5x mirror.
Special device is quite difficult. There is no option to swap ssd drives to bigger one.
When I calculate with a ratio of 0.05 there is not enough space. Also I really need about 500-600GB for pve system and VM system partitions.

Already used special devices with a PBS. GC is so much faster. Idk if my performance will be that poor when using about 15 VM (Windows Server) data disks without special devices. Most metadata should be in cache with ARC?

 

[UdoB]

if my performance will be that poor when using about 15 VM (Windows Server) data disks

That depends on the data access pattern ;-)

The system drive "C:" will be somewhere else? I would not expect that 15 Windows Servers can run entirely from rotating rust - nowadays. Disclaimer: I am really happy to have ZERO experience with that!

Most metadata should be in cache with ARC?

Yes. Also without SD, of course. Actually you can control that by setting ZFS properties:

~# zfs get all  rpool | grep cache
rpool  primarycache          all                       default
rpool  secondarycache        all                       default

~# zfs set primarycache=asdfasdf rpool
cannot set property for 'rpool': 'primarycache' must be one of 'all | none | metadata'

Note that writing data will write meta data too. Adding a SD of any kind will reduce the slow but required physical head movements drastically.

 

[cdn123]

Sure! System partition=C:
they are stored at ssd drives.

What I can change with primary and secondary cache?
Edit: maybe just what is cached in ARC?

 

[UdoB]

What I can change with primary and secondary cache?

Please see man zfsprops for a short description. You'll find a ton of interesting information there.

 

[cdn123]

Primary= ARC
Secondary=L2ARC