Storage with 24 disk per server
- Thread starter Icefire
- Start date
RAIDZ2 is good for fault tolerance, notsomuch for running interactive VMS. On my 14-disk ZFS DRAID, VM GUI response was ass bc it was generating (14) spindles of I/O for everything. Moved the VM(s) to a single SSD and it was fine.
Test your VM realtime response. You may want to reconfigure as a mirror pool - or several, especially if you can invest in larger disks. 4x16TB drives in a mirror pool - (or 6x12TB, if you want ~30TiB of usable free space before compression) - is going to use a lot less electricity over time.
https://wintelguy.com/zfs-calc.pl
Test your VM realtime response. You may want to reconfigure as a mirror pool - or several, especially if you can invest in larger disks. 4x16TB drives in a mirror pool - (or 6x12TB, if you want ~30TiB of usable free space before compression) - is going to use a lot less electricity over time.
https://wintelguy.com/zfs-calc.pl
12-16TB disk rebuilt time will probably end in a lost of the second drive no? its not much fault tolerant
Previously we were using ZFS RAIDZ2 on 6x10TB SAS (7200 rpm) drives for PBS in our setup. All was fine until we tried to restore some ~1TB VMs from this PBS ZFS pool - the read speed of RAIDZ2 is horrible, even with ZFS special device (made of mirrored SSD) attached to it. The read performance of RAIDZ (or classic RAID5/6) is a known pain and not suitable when you need to restore large amounts of data quickly. It might be OK if you have lots of time given to your restoration task, but usually that's not the case.
So we switched these ZFS PBS pools over to ZFS mirrored pool (3 mirrors of 2xDisks each), topped with ZFS special device residing on 2xPCI-E NVME drives. Both read/write speed increased a lot and I/O delay also dropped. You should consider this when thinking about RAIDZ setups.
So we switched these ZFS PBS pools over to ZFS mirrored pool (3 mirrors of 2xDisks each), topped with ZFS special device residing on 2xPCI-E NVME drives. Both read/write speed increased a lot and I/O delay also dropped. You should consider this when thinking about RAIDZ setups.
For both, PVE running VMs and PBS with its millions over millions of small chunk files you need lots of IOPS performance. HDDs are terrible at that. So to make most out of it you probably want to use a 20 HDD striped mirror + 1 SSD SLOG + 2 SSD special device mirror + 1 (or even more and less data HDDs) HDD hot spare and even then a single 10€ SSD will be way faster than those 24 10K HDDs combined.
Or if you want to combine the performance of a raid10 with the reliability of a raidz2 maybe even a striped 3-disk mirror. So 18 HDDs as striped 6x 3-disk mirrors + 3 SDDs as 1x 3-disk mirror as special devices + 2 SSDs as mirrored SLOG + 1 HDD as hot spare.
Or if you want to combine the performance of a raid10 with the reliability of a raidz2 maybe even a striped 3-disk mirror. So 18 HDDs as striped 6x 3-disk mirrors + 3 SDDs as 1x 3-disk mirror as special devices + 2 SSDs as mirrored SLOG + 1 HDD as hot spare.
Last edited: