RAID storage advice please

[barrydocks]

I am rebuilding my office server. I would like to host it as a VM on Proxmox. Currently I am running a software RAID10 with 4x1TB disks and 16GB RAM
I need good read/write performance with some redundancy.
I was thinking of adding an additional drive for RAID6 but I understand the performace is not great. The other option is to use ZFS and add some more RAM?

I would be grateful for some opinions please

 

[LnxBil]

software raid10, despite the technology mdadm or ZFS, will perform better with more RAM for caching and will also perform better with more disks. RAID5 and RAID6 will always be slower than RAID10.

Are those disks SSDs or spinning rust? If they are spinning rust, you may get the best experience with ZFS if you add two metadata enterprise SSDs (e.g. 128G) and you add an 16G optane nvme SLOG device.

 

[barrydocks]

software raid10, despite the technology mdadm or ZFS, will perform better with more RAM for caching and will also perform better with more disks. RAID5 and RAID6 will always be slower than RAID10.

Are those disks SSDs or spinning rust? If they are spinning rust, you may get the best experience with ZFS if you add two metadata enterprise SSDs (e.g. 128G) and you add an 16G optane nvme SLOG device.

Thanks for your reply. Current disks are not SSDs. Would adding a larger SSD help? Would it be worth adding more RAM?

 

[czechsys]

4x1 TB SSDs arent't costly.

 

[LnxBil]

Would adding a larger SSD help?

For the setup I described? Sure, yet 128G is a good minimum for metadata. If you want to store addition stuff besides metadata (so with special_small_blocks) that'll work too. Be sure to buy enterprise SSDs (used ones are totally fine for SOHO environment).

Also consider replacing the harddisks with SSDs as @czechsys pointed out. I bought 960 GB Enterprise Samsung for about 35 euro a piece.

Would it be worth adding more RAM?

It's always worth adding more RAM if you only have 16 GB - my phone has already half as much as your host.

 

[barrydocks]

For the setup I described? Sure, yet 128G is a good minimum for metadata. If you want to store addition stuff besides metadata (so with special_small_blocks) that'll work too. Be sure to buy enterprise SSDs (used ones are totally fine for SOHO environment).

Also consider replacing the harddisks with SSDs as @czechsys pointed out. I bought 960 GB Enterprise Samsung for about 35 euro a piece.


It's always worth adding more RAM if you only have 16 GB - my phone has already half as much as your host.

Thanks for your help. I have found a good deal on some enterprise 1TB SSDs so will swap out the 4 HDDs and increase the RAM to 32GB (max for mainboard). Is is still worth adding a SLOG drive as well? I could only add a single PCIe SLOG drive as there are only 2 free slots on the main board and I want to use on for an additional network card, the only other option is to add a SLOG drive using a SATA interface but I don't thing the Optane devices are available as a SATA drive?

 

[LnxBil]

Is is still worth adding a SLOG drive as well?

Depends on your workload. You will notice it on synchronous writes, yet only there.

I could only add a single PCIe SLOG drive as there are only 2 free slots on the main board

There are also multi-M.2 pci boards available for a single slot. Those use pcie bifurcation and may work on your board, please check first.

only other option is to add a SLOG drive using a SATA interface but I don't thing the Optane devices are available as a SATA drive?

The NVMe Optane has the advantage of fast write times in the microsecond range and that is the performance booster. SATA SLOG may help with spinning rust disks.

 

[barrydocks]

Depends on your workload. You will notice it on synchronous writes, yet only there.


There are also multi-M.2 pci boards available for a single slot. Those use pcie bifurcation and may work on your board, please check first.


The NVMe Optane has the advantage of fast write times in the microsecond range and that is the performance booster. SATA SLOG may help with spinning rust disks.

Thank you for your advice.
I don't think my HPE Gen10 microserver will support a dual NVMe card, there are lots of forum posts around suggesting it's problematic unless you have the Gen10 plus.
So I have picked up a lightly used HPE G110 Gen10 Xenon server to replace the Gen10 Microserver, and now have plenty of PCIe slots, more processing cores, more RAM and more drive bays
Plan is to use:
2x256MB SSD SATA for the OS
4x980MB SSD SATA for storage if possible
2x64GB Optane PCIe drives as SLOG devices.

So, when installing PVE, how are the SLOG devices setup? Or is that something I have to do manually afterwards?

 

[LnxBil]

2x256MB SSD SATA for the OS
4x980MB SSD SATA for storage if possible

Having seperate install from data has only one advantage in my book: beeing able to reinstall withouth loosing data. The OS disks will be used 2-3 GB (compressed ZFS) and the rest is useless. The SSD speed is also only useful on boot.

2x64GB Optane PCIe drives as SLOG devices.

If you haven't bought the Optane already, you don't need to much Optane. The SLOG will only store 5 seconds of data, so 64 GiB / 5 sec = 12,8 GiB / sec. So having 16 GiB is enough. I also partitioned mine to also store second level swap (first level is zswap, in memory compressed-swap).

So, when installing PVE, how are the SLOG devices setup? Or is that something I have to do manually afterwards?

No, the PVE install will only setup a simple ZFS pool. You can attach the SLOG device later.

 

[barrydocks]

Having seperate install from data has only one advantage in my book: beeing able to reinstall withouth loosing data. The OS disks will be used 2-3 GB (compressed ZFS) and the rest is useless. The SSD speed is also only useful on boot.

OK, so I could get away with smaller drives for the OS? Then, could I also partition them and use some of the space for metadata as you mentioned in an earlier post? Or not bother and put everything on the 4x980MB SSDs?

 

[LnxBil]

OK, so I could get away with smaller drives for the OS?

Yes, but keep in mind that all SSDs should be enterprise class sata or sas. The smallest disk I can think of are 128 GB for 20 euro (used).

Then, could I also partition them and use some of the space for metadata as you mentioned in an earlier post?

The metadata split makes only sense if you have different classes of drives, e.g. harddisks and (enterprise) SSD in the same pool.

Or not bother and put everything on the 4x980MB SSDs?

I would put everything on the big SSDs and skip the OS SSDs, yet this is MY decision. There are a lot of people always splitting this. I'm comfortably solving low level ZFS problems and if you don't want to work on that level on a problem, you may keep the seperate boot pool so that you can reinstall PVE without touching the actual data. Keep in mind that the VM configuration will still be on the OS drives, so that you will loose the configuration unless backuped up or recovered.

 

[barrydocks]

Yes, but keep in mind that all SSDs should be enterprise class sata or sas. The smallest disk I can think of are 128 GB for 20 euro (used).


The metadata split makes only sense if you have different classes of drives, e.g. harddisks and (enterprise) SSD in the same pool.


I would put everything on the big SSDs and skip the OS SSDs, yet this is MY decision. There are a lot of people always splitting this. I'm comfortably solving low level ZFS problems and if you don't want to work on that level on a problem, you may keep the seperate boot pool so that you can reinstall PVE without touching the actual data. Keep in mind that the VM configuration will still be on the OS drives, so that you will loose the configuration unless backuped up or recovered.

Excellent. To keep things simple and the cost down, I'll go with everything on the 4x1TB enterprise SSDs and 2x16GB Optane as SLOG devices.

Thanks you so much for your help and advice!

 

[barrydocks]

Yes, but keep in mind that all SSDs should be enterprise class sata or sas. The smallest disk I can think of are 128 GB for 20 euro (used).


The metadata split makes only sense if you have different classes of drives, e.g. harddisks and (enterprise) SSD in the same pool.


I would put everything on the big SSDs and skip the OS SSDs, yet this is MY decision. There are a lot of people always splitting this. I'm comfortably solving low level ZFS problems and if you don't want to work on that level on a problem, you may keep the seperate boot pool so that you can reinstall PVE without touching the actual data. Keep in mind that the VM configuration will still be on the OS drives, so that you will loose the configuration unless backuped up or recovered.

Ok now have server set up with 4x1TB SSD and 2x16GB Optane drives. I have installed PVE with ZFS on the 4x1TB SSDs. How do I add the 2x16GB Optane drives as SLOG?

 

[LnxBil]

Ok now have server set up with 4x1TB SSD and 2x16GB Optane drives. I have installed PVE with ZFS on the 4x1TB SSDs. How do I add the 2x16GB Optane drives as SLOG?

zpool add [pool name] log mirror /dev/nvme[x] /dev/nvme[x2]

 

[barrydocks]

zpool add [pool name] log mirror /dev/nvme[x] /dev/nvme[x2]

Awesome, thank you