ZFS pool for new proxmox installation

[AT_WL82]

Hey guys, New on this team and journey with proxmox
Having the issue and play a little bit on it trying to discover how to setup a RAIDZ1 with
3x 2TB HDD
Im trying to to do a RAIDZ1 that allow me to have 1 failure and the rest for data
Proxmox just give option for RAID0 with total of 6TB

Any idea how to create it I do not see RAIDZ1 option on the list

Also which is the better compression in terms of data speed performance

And last question If allows me to create a ZFS pool with HDDS of different sizes

Thanks in advance

Using version 7.4 cause 8.0 does not install on my server is a DELL R610 trying to upgrade proxmox later

 

[Dunuin]

Also which is the better compression in terms of data speed performance

ZSTD for bad performance but good compression ratio. LZ4 for very good performance and ok compression ratio. I would stick with the default LZ4 unless the data is well compressible, rarely accessed and you got tons of CPU power you don't need.

Having the issue and play a little bit on it trying to discover how to setup a RAIDZ1 with

Should definitely be in the installer as raidz1 or raid5. Make sure to increase the "block size" of the ZFS storage from 8K to 16K before creating your first VM or you will waste another 1TB of capacity.
Also keep in mind that having 4TB available doesn't mean you should use all of it. You usually don't want to fill it too much as it will fragment faster and become slow. I personally always set a 90% quota (so 10% won't be usable and can't be filled by accident) and try to be below 80%.
So more like 3TiB of real usable capacity. And if you want to use snapshots even less for actual data, as the snapshots will grow and also consume space of those 3TiB. So probably more like 2TiB for data if you want 33% for snapshots.

And last question If allows me to create a ZFS pool with HDDS of different sizes

Yes, but don't mix different sized disks per vdev or the bigger disks will be limited to the size of your smallest disk of that vdev. So for example 3x 2TB raidz1 + 3x 4TB raidz1 would be fine and give you 12TB but a single 6 disk raidz1 using the same disks would only give you 10TB.

 

[JensF]

ZSTD for bad performance but good compression ratio.

I hope you can prove this. All test I read was same performance as LZ4 but a better compression.

 

[Dunuin]

I hope you can prove this. All test I read was same performance as LZ4 but a better compression.

I see 8 of 8 CPU threads at 100% utilization and way worse throughput when writing dozens of GBs of data to my ZSTD datasets. Nothing I would use expect for cold storage, especially not when using SSDs. With LZ4 the HDDs are bottlenecking. With ZSTD it's always the CPU.

Also annoying when all the other guests are slowed down, while doing a lot of IO, because ZFS is stealing all the CPU power.

So it really depends on how beefy your CPU is. Here, most of the times, on all servers, the saved space isn't worth the additional CPU load.

 

[JensF]

I see 8 of 8 CPU threads at 100% utilization and way worse throughput when writing dozens of GBs of data to my ZSTD datasets. Nothing I would use expect for cold storage, especially nkt when using SSDs. With LZ4 the HDDs are bottlenecking. With ZSTD it's always the CPU.

I've never seen such behavior in our system. Moved some gigabytes from one zvol to another to find the best volblocksize for our mirroset of two PM1735...

 

[AT_WL82]

ZSTD for bad performance but good compression ratio. LZ4 for very good performance and ok compression ratio. I would stick with the default LZ4 unless the data is well compressible, rarely accessed and you got tons of CPU power you don't need.



Should definitely be in the installer as raidz1 or raid5. Make sure to increase the "block size" of the ZFS storage from 8K to 16K before creating your first VM or you will waste another 1TB of capacity.
Also keep in mind that having 4TB available doesn't mean you should use all of it. You usually don't want to fill it too much as it will fragment faster and become slow. I personally always set a 90% quota (so 10% won't be usable and can't be filled by accident) and try to be below 80%.
So more like 3TiB of real usable capacity. And if you want to use snapshots even less for actual data, as the snapshots will grow and also consume space of those 3TiB. So probably more like 2TiB for data if you want 33% for snapshots.



Yes, but don't mix different sized disks per vdev or the bigger disks will be limited to the size of your smallest disk of that vdev. So for example 3x 2TB raidz1 + 3x 4TB raidz1 would be fine and give you 12TB but a single 6 disk raidz1 using the same disks would only give you 10TB.

Really Appreciate it, Understood. I learn the availability of the RAID on the initial installer but I want to create the ZFS pool later after installation I ended put in the proxmox system in a 64gb USB and the my idea is to use those 3x2TB HDD for the ZFS just for information but with the ability to grow it later as I get more hdds i have also 2 NVME of 2 TB that i create a mirror with just for the running of the VMs because the performance of speed on this hdds, let me know if is a good idea, but my problem is creating the ZFS pool for data but with the option to add more hdd later and increase capacity but at same same have at list 1 disk of failure or 2 depend on increase. Thanks again

 

[AT_WL82]

I see 8 of 8 CPU threads at 100% utilization and way worse throughput when writing dozens of GBs of data to my ZSTD datasets. Nothing I would use expect for cold storage, especially not when using SSDs. With LZ4 the HDDs are bottlenecking. With ZSTD it's always the CPU.

Also annoying when all the other guests are slowed down, while doing a lot of IO, because ZFS is stealing all the CPU power.

So it really depends on how beefy your CPU is. Here, most of the times, on all servers, the saved space isn't worth the additional CPU load.

So I can not comment at all hahah. which will be best choice base on your experiences

 

[AT_WL82]

ZSTD for bad performance but good compression ratio. LZ4 for very good performance and ok compression ratio. I would stick with the default LZ4 unless the data is well compressible, rarely accessed and you got tons of CPU power you don't need.



Should definitely be in the installer as raidz1 or raid5. Make sure to increase the "block size" of the ZFS storage from 8K to 16K before creating your first VM or you will waste another 1TB of capacity.
Also keep in mind that having 4TB available doesn't mean you should use all of it. You usually don't want to fill it too much as it will fragment faster and become slow. I personally always set a 90% quota (so 10% won't be usable and can't be filled by accident) and try to be below 80%.
So more like 3TiB of real usable capacity. And if you want to use snapshots even less for actual data, as the snapshots will grow and also consume space of those 3TiB. So probably more like 2TiB for data if you want 33% for snapshots.



Yes, but don't mix different sized disks per vdev or the bigger disks will be limited to the size of your smallest disk of that vdev. So for example 3x 2TB raidz1 + 3x 4TB raidz1 would be fine and give you 12TB but a single 6 disk raidz1 using the same disks would only give you 10TB.

I get it. thanks i will go for it, thanks for your time

 

[Dunuin]

I ended put in the proxmox system in a 64gb USB

Thats bad. PVE will kill USB pen drives and SD cards in no time. Only use proper HDDs or SSDs that can handle the writes.

 

[AT_WL82]

Thats bad. PVE will kill USB pen drives and SD cards in no time. Only use proper HDDs or SSDs that can handle the writes.

ok. thanks
Question
Will be ok if use a MSATA hdd trough a USB adapter? do this will work ok?

 

[Dunuin]

Have a look at the datasheet. If the TBW of the mSATA SSD looks comparable to other SSDs and it got wear leveling then it should be fine.

 

[AT_WL82]

Have a look at the datasheet. If the TBW of the mSATA SSD looks comparable to other SSDs and it got wear leveling then it should be fine.

Thanks again
Other questions are
I do not see the ability to create a ZFS pool with RAID Z1 after installation it just don't give me the option on the creation wizard I want to have with 3x2TB HDD the option to have one for failover and rest for data so ZFS should be around 4TB, but also to have the opportunity to keep growing later as I do on synology, do we had that option here with proxmox?
What will be a good size for the proxmox HDD to don't have the necessary to be upgrading later ?
promox and VMs I read does not need to be in same hdd right?,
my point is I want to have proxmox in a hdd along of the data for the VMs

 

[Dunuin]

I do not see the ability to create a ZFS pool with RAID Z1 after installation

Its there at "YourNode -> Disks -> ZFS -> Create: ZFS -> RAID Level" where you can chose "RAIDZ" which is Raidz1/Raid5.

so ZFS should be around 4TB, but also to have the opportunity to keep growing later as I do on synology, do we had that option here with proxmox?

No, you can't extend a raidz yet. Whats working is buying 3 more disks and adding them to your pool as another raidz1 vdev.

What will be a good size for the proxmox HDD to don't have the necessary to be upgrading later ?

For PVE itself 32GB should be plenty. If you need to store other stuff like ISOs and templates there of cause more.

promox and VMs I read does not need to be in same hdd right?,

Yes, not required and even recommended to not have them both on the same disks in case you got the disks and slots to separate it.

 

[AT_WL82]

Its there at "YourNode -> Disks -> ZFS -> Create: ZFS -> RAID Level" where you can chose "RAIDZ" which is Raidz1/Raid5.


No, you can't extend a raidz yet. Whats working is buying 3 more disks and adding them to your pool as another raidz1 vdev.


For PVE itself 32GB should be plenty. If you need to store other stuff like ISOs and templates there of cause more.


Yes, not required and even recommended to not have them both on the same disks in case you got the disks and slots to separate it.

Thank you so much, hope I can help you with something different in life. No proxmox this time haha. I just saw that when create the ZFS wit the 3x2TB with RAIDZ the pool is the total 6TB and not 4tb as expected that do this work here for 1hdd on redundancy, I will need to study better zfs honestly