Should i install proxmox on my 4TB SSD RAID 5 setup ?

[moda20]

I am moving my homelab from docker containers to proxmox VE VMs. my motivation is to be able to backup whole folders / images more easily and not have 5 or 6 different postgres containers at the same time. A secondary goal would be to learn mopre about virtualization (Okey knowledge).

My goal is to have a stable drive to use. i had prior issues with bad disks failing and breaking my filesystem. i am going to get 3 2TB SSDs and put them in RAID 5 with 4Tb of usable space.

I want to know Hoiw can i install it on the RAID setup and if the RAID config is created by the installer or should i have it ready somehow.
the second question is shoudl i do the isntall on the full drive ? knowing that i don't have any other SSDs and my HDDs are nor fast enough to be used for other than bulk storage and maybe backups and temporary downloaded files.

Thanks in advance.

 

[d.oishi]

If you are using a real hardware RAID controller and you want to use hardware RAID, then the RAID5 array should be configured in the RAID controller firmware first. During the Proxmox installation, that array should appear as a single logical disk.

In that case, I would not put ZFS on top of the hardware RAID array. Just install Proxmox on the logical disk using the normal non-ZFS installer option, for example ext4/xfs with LVM-thin.

If you are not using a hardware RAID controller, or if your controller can expose the disks individually, for example in JBOD/HBA/IT mode, then you can let the Proxmox installer create a ZFS RAIDZ-1 pool using the three SSDs.

Since you only have three SSDs, it is fine to use all three of them for the Proxmox installation and VM storage in the same ZFS pool. The installer will handle the system and VM storage layout on that pool.

However, RAIDZ-1 can tolerate one disk failure.

This manual section may also help:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#installation_installer

 

[leesteken]

RAIDz1 is terrible for VMs and drives without PLP are also no fun with ZFS. Hardware RAID5 (with battery backup for sync write caching) without ZFS is fine.
https://forum.proxmox.com/threads/fabu-can-i-use-zfs-raidz-for-my-vms.159923/ (fixed, thanks to @Johannes S)

 

[alexskysilk]

RAIDz1 is terrible for VMs and drives without PLP are also no fun with ZFS.

words like "terrible" are terrible. they provide no context. You could say "RAIDz1 performs poorly compared to RAID10" or "RAIDz1 provides poor risk management vs RAIDz2" but without knowing the use case and expected performance and fault tolerance requirements any absolute judgment fails.

With regards to plp- zfs handles power loss as well as any other file system. you might just lost a second of pending writes. Do you want it? yes. do you need it? same question as above...

Hardware RAID5 (with battery backup for sync write caching) without ZFS is fine.

The biggest issue with RAIDz1 is single parity; RAID5 buys you nothing. PLP obviates the need for a BBU.

 

[UdoB]

With regards to plp- zfs handles power loss as well as any other file system.

Yes.

you might just lost a second of pending writes.

Up to more than five seconds - that's the default interval for writing out (async-) buffered content. More than five because there are two Transaction Groups "TXG": one is active and receives data to be written (from a program) while the other is being emptied by writing its data to the final destination.

But this (power loss) is only one aspect of having PLP. The other one has the same importance for me: it makes sync writes multiple times faster. The reasoning has been discussed multiple times. Cheap SSD w/o PLP under load may feel slower than rotating rust. Yes. Been there, felt that ;-)

The third aspect is durability: with PLP the same SSD would work much, much longer as the actual writes of the internal groups-of-cells happens less often and can get sorted in a more clever way. Remember that these groups alway needs to get read(!) completely, some content will get modified and then the whole shebang has to be written back to where it belongs. That's slow by definition..., and the Copy-on-Write approach of ZFS does not help too...

 

[leesteken]

words like "terrible" are terrible. they provide no context.

The problem with RAIDz1 is the lack of (read) IOPS compared to mirrors while putting multiple machines worth of IO to it (and often less than expected usable space). I added a link with more detailed discussing and the "terrible" was just my (opinionated) summary. Unfortunately, the link was broken but this has been fixed.

 

[alexskysilk]

Up to more than five seconds

true.

it makes sync writes multiple times faster.

If you want to buy an 18 wheeler for haul groceries thats your call- it doesnt mean you need an 18 wheeler. match the solution to the problem. Not every usecase benefits.

The third aspect is durability: with PLP the same SSD would work much, much longer as the actual writes of the internal groups-of-cells happens less often and can get sorted in a more clever way

Thats an argument for a newer drive, not for a plp. technology doesnt stand still.

The problem with RAIDz1 is the lack of (read) IOPS compared to mirrors while putting multiple machines worth of IO to it (and often less than expected usable space).

See that makes a lot more sense

 

[J-Rod]

For home lab, it's whatever you want, or need, I suppose. But with software RAID, for simpler troubleshooting and recovery from faults, I would lean towards a mirrored array for boot, rather than single parity, dual parity or even RAID10.

So consider sourcing a fourth SSD so you can set up two mirrored arrays.

 

[moda20]

If you are using a real hardware RAID controller and you want to use hardware RAID, then the RAID5 array should be configured in the RAID controller firmware first. During the Proxmox installation, that array should appear as a single logical disk.

In that case, I would not put ZFS on top of the hardware RAID array. Just install Proxmox on the logical disk using the normal non-ZFS installer option, for example ext4/xfs with LVM-thin.

If you are not using a hardware RAID controller, or if your controller can expose the disks individually, for example in JBOD/HBA/IT mode, then you can let the Proxmox installer create a ZFS RAIDZ-1 pool using the three SSDs.

Since you only have three SSDs, it is fine to use all three of them for the Proxmox installation and VM storage in the same ZFS pool. The installer will handle the system and VM storage layout on that pool.

However, RAIDZ-1 can tolerate one disk failure.

This manual section may also help:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#installation_installer

I am only looking at software RAID for now unless i can get something already ready OOB (drives and cards or a ready enclosure that i can slot in), but i ma not seeing how i can find that rn xD

I see, 3 drives raid 5's 1 drive redundency is more a short term solution until i can buyu more SSDs cheaply, hopefully by the end of next year.

I understand the low read IOPS given by raid 5 compared to a mirrored drives. my existing setups does have larger-than-memory-sized databases with heavy aggregations and queries done regularly so more IOPS would be nice, but the prices don't help me especially when this is still a homelab and i am still trying a bunch of stuff to know exactly what i need.

RAIDz1 is terrible for VMs and drives without PLP are also no fun with ZFS. Hardware RAID5 (with battery backup for sync write caching) without ZFS is fine.
https://forum.proxmox.com/threads/fabu-can-i-use-zfs-raidz-for-my-vms.159923/ (fixed, thanks to @Johannes S)

I can still get a home system UPS and mitigate most power loss issues (PLP and BBU not being present on the raid setup), which is something i intend to do as well. my initial fs corruption problem was triggered by a recurrent power outage and a lowend SSD controller.

For home lab, it's whatever you want, or need, I suppose. But with software RAID, for simpler troubleshooting and recovery from faults, I would lean towards a mirrored array for boot, rather than single parity, dual parity or even RAID10.

So consider sourcing a fourth SSD so you can set up two mirrored arrays.

Yes, a fourth SSD is needed for the failure incident when only 3 drives are active. but again i am trying to bring back my homelab with the minimal cost and enough redundency, with about 4tb of space

I don't want to take this thread too out of scope, but would DRAM-less SSDs like WD blue make a difference compared to SSDs with DRAM when used in a RAID 5 with ZFS and for VM storage ?

 

[leesteken]

I can still get a home system UPS and mitigate most power loss issues (PLP and BBU not being present on the raid setup), which is something i intend to do as well. my initial fs corruption problem was triggered by a recurrent power outage and a lowend SSD controller.

The ability to cache/delay sync writes (and the performance improvement that comes with it) is not possible with a UPS. The drive itself needs to know that it can delay writes safely. Otherwise, it cannot and you have to wait for the data to be actually written (regardless of UPS). The problem with consumer SSDs is that they rewrite data (from SLC cache to QLC for example, or combining fragments of trimmed data) in the background and can cause corruption on power loss because of it. This you can prevent with a UPS but it does not bring the same performance of PLP.

EDIT: everything you can think of is probably already discussed on this forum, but sometimes hard to find. The consensus is: you need PLP for happy performance with ZFS and you then also don't need to worry about wear on the SSD. Lots of people cannot believe they wasted money on consumer drives and it's never a happy conversation trying to explain it to them when they experience bad performance. Nobody here complains about (second-hand) enterprise SSDs with PLP, only people who thought they had a good deal with cheap consumer/QLC drives.

 

[uzumo]

It might actually be better to have a DRAM cache than a DRAM-less solution that uses the host’s memory as an HMB.

The main point of discussion above wasn’t that PLP is used to protect against data loss, but rather whether you actually need an enterprise SSD for your home lab that features PLP—one that returns immediate ACKs during I/O operations and doesn’t require an SLC cache.

That said, the most important factor is whether the volume of data being written is at a level that can be handled by a consumer-grade SSD.

At worst, when I/O exceeding the SLC cache occurs, I/O delay will spike sharply, causing all virtual machines to halt simultaneously.

Rest assured that this won’t happen as long as you absolutely avoid such write operations. Of course, I have absolutely no idea how much data is okay.

*When a GC occurs, the system reads the area previously occupied by the SLC cache and moves it to TLC or QLC storage. As a result, the system is unable to return an ACK, causing I/O latency to spike to high values, such as 100%.

If you are performing backups or similar operations, I believe there is a possibility that you may exceed the capacity of the SLC cache. While this is certainly a challenge, you’ll need to consider your options—whether to accept it as unavoidable or to carefully set bandwidth limits for each virtual machine and backup, for example.

If you want to use it without worrying about anything, you need to choose a high-quality tool—even if its features are a bit excessive.
If you’re only using it for very light tasks, or if you don’t mind going through a hell of a hassle, the current options are sufficient.

If I were to buy one now, I don't think there would be much of a price difference between a used 3.72TB enterprise-grade SSD and a brand-new 4TB consumer-grade SSD, would there?

Of course, even if people say it’s overkill, I still recommend an SSD with PLP. Eventually, the time you spend on frequent replacements and troubleshooting will exceed the purchase price.

There's usually this much of a difference in base speed.

・U.2 KIOXIA KCD6XLUL960G

root@pve2:~# pveperf

HD SIZE:           860.53 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND:     16415.25 ★


・SE900NVG3 2048G

root@pve2:~# pveperf /test

HD SIZE:           1845.00 GB (test)
FSYNCS/SECOND:     958.31 ★

 

[J-Rod]

Back in 2015-2016 I spent a few thousand USD on a pile of used enterprise SSDs with 10 DWPD ratings and put them to work for all my 24x7x365 storage needs. DVR, NVR, caching, VMs, etc. Even a decade later they are all still above 95% life remaining. Money well spent.