System storage concept questions

[jeschke]

Hi all,

I finally have the time to setup my new proxmox-server, but as I am new to ZFS I need to understand how a system is setup best before I order the relevant storage devices!
Many posts in the forum discuss hardware requirements but I still have questions when it comes to redundant system setups with NVMe's:

My server:
ASUS P11C-M/4L, 32GB ECC-RAM, i3-8300

Use case:
- PiHole
- pfSense
- NC
- FreeNAS
- different containers for testing
- possibly webservers
- possibly mailserver
- ...

Planned storage setup:
PVE system: 2x small SATA-SSD's (ZFS mirror) - for example Samsung 860 EVO/Pro
VM's/CT's: 2x 1TB m.2 NVMe SSD's (ZFS mirror) - for example Samsung 970 EVO/Pro
Data/NAS: several HDD's (ZFS Raid) - probably WD red

Open questions:
1. Does the above setup (especially the NVMe's) make sense when it comes to performance and redundancy or are single devices with a proper backup-schedule a better solution?
2. What kind of SSD's/NVMe's are recommended, are Samsung EVO/Pro a good choice? I have read about SSD's being worn out fast when used as system storage device
2. What about SSD-caching, would it improve performance with my setup or are 32GB ECC enough for my use case?

Please be patient as I am new to proxmox and especially ZFS!

Thanks a lot in advance!!
Jeschke

 

[guletz]

Hi,

1. It is ok. But you need to reduce the write io if you want to keep a long life for your ssd/nvme. You could move some folders on hdd pool (like /tmp, /var/log)

2. as a general rule DataCenter ssd are more reliable and will live long

3. you need ssd caching only for hdd pool only. but you can start without it and then you will see if you really need or not.

Good luck / Bafta

 

[jeschke]

Hi guletz,

thanks a lot for your reply!

I will probably go with the follwing (for a start):
- 2x Intel DC P4511 for OS and VM's (ZFS mirrored)
- 2x WD Red 10TB for data (ZFS mirrored)
...and later add a cache-SSD if necessary

Furthermore I will follow your advice and move some folders on hdd pool (like /tmp, /var/log) .

In the end I want a system that I can leave alone (hardware) for the next years without too much maintenance. I hope this setup does the trick

Cheers,
Christian

 

[tom]

2x WD Red 10TB for data (ZFS mirrored)

this will be deadly slow without a fast SSD cache device.

 

[jeschke]

this will be deadly slow without a fast SSD cache device.

Hi Tom,

I am trying to find out about a suitable setup for days now, please advice on the following setup:
- 2x 1TB Intel DC P4511 for OS and VM's (ZFS mirrored)
- 2x 10TB WD Red for data (ZFS mirrored)
- 1x Intel DC S4500 for Cache, but what size?!

Thanks in advance!
Jeschke

 

[guletz]

2x WD Red 10TB for data (ZFS mirrored)

Hi

Be aware, you could see that yours hdd can be broken at the same time. For any hdd > 4 TB I would go with a raidz2. Recently I see this on a raidz2 with 4 x 8 TB when 2 hdds was decide to die at the same time

Good luck / Bafta

 

[guletz]

- 1x Intel DC S4500 for Cache, but what size?!

128-256 GB is ok. You must understand that any data will be in zpool cache on ssd will eat your RAM also(4 bytes for each block stored on ssd if I remember)

 

[tom]

Intel DC S4500

outdated model, not really the fastest one. but maybe fast enough for your workload.

 

[jeschke]

outdated model, not really the fastest one. but maybe fast enough for your workload.

can you recommend any specific DC SSD?

what about these:
Intel® SSD D3-S4510 - 240GB
Intel® SSD D3-S4610 - 240GB
Intel® SSD DC S3110 - 256GB

And what about Samsung 983 instead of the Intel DC P4511? As they are a lot cheaper...

 

[jeschke]

maybe you can also help me with the decision regarding a new case:

Can anyone give me a recommendation on one of the below? Any other hint for good a case is very welcome!
- Fractal Design Define R5 Black (8x 3,5")
- Nanoxia Deep Silence 4 (6x 3,5")
- SilverStone SST-CS350B (5x 3,5" HotSwap)

Requirements are:
- can fit a micro-ATX board
- as silent as possible
- can handle at least 5 HDD's

Cheers,
Jeschke