Storage with 24 disk per server

Icefire

Member
Mar 20, 2022
18
0
6
43
Hi,
Currently looking to deploy 1 PBS + 2 VE, all of them with around 28TB (24x1.2TB SAS 10k)
I have made pools of 1 pool of 3x 8 disk RaidZ2, giving 6 disk of parity.

What would be best for fault tolerance?
 
RAIDZ2 is good for fault tolerance, notsomuch for running interactive VMS. On my 14-disk ZFS DRAID, VM GUI response was ass bc it was generating (14) spindles of I/O for everything. Moved the VM(s) to a single SSD and it was fine.

Test your VM realtime response. You may want to reconfigure as a mirror pool - or several, especially if you can invest in larger disks. 4x16TB drives in a mirror pool - (or 6x12TB, if you want ~30TiB of usable free space before compression) - is going to use a lot less electricity over time.

https://wintelguy.com/zfs-calc.pl
 
You may want to reconfigure as a mirror pool - or several, especially if you can invest in larger disks. 4x16TB drives in a mirror pool - (or 6x12TB, if you want ~30TiB of usable free space before compression) - is going to use a lot less electricity over time.

https://wintelguy.com/zfs-calc.pl
12-16TB disk rebuilt time will probably end in a lost of the second drive no? its not much fault tolerant
 
Previously we were using ZFS RAIDZ2 on 6x10TB SAS (7200 rpm) drives for PBS in our setup. All was fine until we tried to restore some ~1TB VMs from this PBS ZFS pool - the read speed of RAIDZ2 is horrible, even with ZFS special device (made of mirrored SSD) attached to it. The read performance of RAIDZ (or classic RAID5/6) is a known pain and not suitable when you need to restore large amounts of data quickly. It might be OK if you have lots of time given to your restoration task, but usually that's not the case.

So we switched these ZFS PBS pools over to ZFS mirrored pool (3 mirrors of 2xDisks each), topped with ZFS special device residing on 2xPCI-E NVME drives. Both read/write speed increased a lot and I/O delay also dropped. You should consider this when thinking about RAIDZ setups.
 
  • Like
Reactions: Kingneutron
For both, PVE running VMs and PBS with its millions over millions of small chunk files you need lots of IOPS performance. HDDs are terrible at that. So to make most out of it you probably want to use a 20 HDD striped mirror + 1 SSD SLOG + 2 SSD special device mirror + 1 (or even more and less data HDDs) HDD hot spare and even then a single 10€ SSD will be way faster than those 24 10K HDDs combined.

Or if you want to combine the performance of a raid10 with the reliability of a raidz2 maybe even a striped 3-disk mirror. So 18 HDDs as striped 6x 3-disk mirrors + 3 SDDs as 1x 3-disk mirror as special devices + 2 SSDs as mirrored SLOG + 1 HDD as hot spare.
 
Last edited:
  • Like
Reactions: Kingneutron

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!