zfs installation and selection of disks

[S0mE1]

Hello everyone,

Before I start, I'd like to wish you all a happy new year.

I need your help, I currently have the lenovo m80q gen3 with the following configuration:
- 2 NVME disks of 2to for the OS and for the storage of the virtual machines (WD red nvme sn700) that I will use with zfs in raid 1
- 32GB or 64GB I don't know yet
- A slot for a ssd sata

I have not been able to take enterprise quality disks and I would like to have a system that is reliable, so my questions are:
- I've read the zfs doc which advises using more ram (1gb for each To), do you advise 32 or 64?
- Can I use the remaining SATA disk as a cache? or to limit data writing to the 2 NVMEs? What would you advise me to do with this disk? If so, which disk would you recommend and what size?
- Do you have any advice on how to limit the premature use of my disks? In this doc: https://pve.proxmox.com/wiki/ZFS_on_Linux it advises limiting the ram but do I need to do anything else?

I'd like to point out that for the moment everything is new and that in a few months' time I'm planning to set up a PBS, so I don't want to use this sata disk for backing up vm's.

Thanks for your answers

Matt

 

[Dunuin]

- I've read the zfs doc which advises using more ram (1gb for each To), do you advise 32 or 64?

Depends on what you are planning to run. Here, RAM is usually running out first. So I would make sure to at least be able to add more RAM later.

- Can I use the remaining SATA disk as a cache? or to limit data writing to the 2 NVMEs? What would you advise me to do with this disk? If so, which disk would you recommend and what size?

Cache should be lower latency. So no point to use a slow SATA SSD as a cache for a fast NVMe SSDs. If you want less wear on the NVMes you should have bought some SSDs with power-loss protection in the first place like a Samsung PM9A3/PM983, Micron 7450 Pro or Solidigm P4511 that could cache sync writes in DRAM to optimize data before writing them to NAND for less wear. Would also have increased the IOPS performance of sync writes by multiple magnitudes.
And those Enteprise SSDs aren't more expensive than your SN700 if you for example compare the 3650TB TBW of the 2TB Micron 7450 Pro for 222€ against the 2500TB TBW of the SN700 2TB for 148€. You pay 50% more but also get a 46% higher TBW and waaaay better performance under continuous or sync writes.
Only good excuse to not buy enterprise SSDs would be if you really need those 2TB of capacity and your motherboard isn't supporting 22110 formfactor M.2s.

- Do you have any advice on how to limit the premature use of my disks? In this doc: https://pve.proxmox.com/wiki/ZFS_on_Linux it advises limiting the ram but do I need to do anything else?

Limiting RAM is only required if you run into problems with OOM. Free RAM is wasted RAM, so not a bad idea to let ZFS use all of it if no other guest is using it.

 

[S0mE1]

Depends on what you are planning to run. Here, RAM is usually running out first. So I would make sure to at least be able to add more RAM later.


Cache should be lower latency. So no point to use a slow SATA SSD as a cache for a fast NVMe SSDs. If you want less wear on the NVMes you should have bought some SSDs with power-loss protection in the first place like a Samsung PM9A3/PM983, Micron 7450 Pro or Solidigm P4511 that could cache sync writes in DRAM to optimize data before writing them to NAND for less wear. Would also have increased the IOPS performance of sync writes by multiple magnitudes.
And those Enteprise SSDs aren't more expensive than your SN700 if you for example compare the 3650TB TBW of the 2TB Micron 7450 Pro for 222€ against the 2500TB TBW of the SN700 2TB for 148€. You pay 50% more but also get a 46% higher TBW and waaaay better performance under continuous or sync writes.
Only good excuse to not buy enterprise SSDs would be if you really need those 2TB of capacity and your motherboard isn't supporting 22110 formfactor M.2s.



Limiting RAM is only required if you run into problems with OOM. Free RAM is wasted RAM, so not a bad idea to let ZFS use all of it if no other guest is using it.

Hello Dunuin,

Thank you for your reply.

Yes I know about the discs you mention, but the constraints I have are:
- 2TB per disk
- NVME size 2280
- Budget, because the 2TB 2280 from wd red cost me 130€.

If I understand your answer correctly, are you advising me to stay as I am and not use the sata disk?

As far as ram is concerned, is it possible to set a limit and then increase/decrease it depending on how I use it?

Thank you and have a nice day

 

[Dunuin]

If I understand your answer correctly, are you advising me to stay as I am and not use the sata disk?

I don't see a good use case except for backups which you don't want.

As far as ram is concerned, is it possible to set a limit and then increase/decrease it depending on how I use it?

Yes, thats possible.