Hi!
I have very good user experince with pve + pbs solutions and thanks for the good software! Currently pbs system runs on system like
- server: PRIMERGY RX2530 M7S
- cpu: 2 x Intel GOLD 6548Y+ (core/threads 32/64 per cpu)
- memory: 512 G DDR5
- storage: 10 x 15 TB Micron 7450 nvme devices
- zfs filesystem under datastore (mirrored raidz1)
- network adapter: Mellanox Dual 25Gb/s Ethernet Adapter ConnectX®-6 lx en
I am thinking of expanding and wonder if you would recommend system like
- server: SSG-221E-NE324R Supermicro Storage SuperServer
- cpu: 2 x Intel GOLD 6548Y+ (core/threads 32/64 per cpu)
- memory: 2 TB DDR5
- storage: 24 x 30.72 TB NVMe SSD PCIe 4.0 x4 (<0.5 DWPD) E3.S 7.5mm - Solidigm SSD D5-P5336
- zfs filesystem under datastore (3 x RAIDZ2 (8 drives each))
- network adapter: Mellanox Dual 100Gb/s Ethernet Adapter ConnectX®-6 Dx EN (2x QSFP56), OCP 3.0
I am a little bit worried if it is too much into one chassis, on the other hand it seems also resonable
. it has lots of storage divices directily connected to cpu lines i.e. without extra adapters
. solidigm ssd d5-p5336 are not maybe fastest nvme devices and they are pcie 4.0 etc but still this is nvme (spec says then read 7000 MBait/s and write 3000 MBait/s)
. i dont expect to have very-very busy backuping going on i.e. i dont expect to run info disk dwpd limit) - for say ceph i would maybe consider more performant drives write-wise)
. 30 TB is big maybe in sense if they fail then replacement takes about 3 hours to copy (considering speed on 3 GByte/s and there is 30 000 GByte to go)
. 30 TB vs 15 TB seems big change but cost where i looked at it is not linear i.e. it is only fragment higher than the other (also disk performance is not so good)
. it has nice compute resource in terms of cpu and memory
. i understand general recommendation for zfs is 2-4 G memory per 1 TB of usable data: 24 x 30 is 760 raw, say 1/4 is lost because of redudancy so 540 TB remains as usable: and it is about 4 * 540 equals about 2 TB of memory
. network is fast (connected to Mellanox sn2100 switch which has splittable 100 Gbit/s ports)
. i expect zfs trimming to happen also which i think generally is a good thing
. unfortunately this server is not listed at https://ubuntu.com/certified/servers?q=supermicro&category=Server
. unfortunately for this server is ubuntu not listed at https://www.supermicro.com/en/suppo...ration=X13&form_factor=2U&category=Storage+DP
Also, i looked into other manufacturers offers, Lenovo seems to have SR650 for example, and others too but this supermicro seems most interesting. (i found that is profitable to ask chatgpt 'what lenovo computer is comparable to supermicro SSG-221E-NE324R model' etc, right
I would be very thankful if someone would put their insight into this endeavour!
Best regards,
Imre
I have very good user experince with pve + pbs solutions and thanks for the good software! Currently pbs system runs on system like
- server: PRIMERGY RX2530 M7S
- cpu: 2 x Intel GOLD 6548Y+ (core/threads 32/64 per cpu)
- memory: 512 G DDR5
- storage: 10 x 15 TB Micron 7450 nvme devices
- zfs filesystem under datastore (mirrored raidz1)
- network adapter: Mellanox Dual 25Gb/s Ethernet Adapter ConnectX®-6 lx en
I am thinking of expanding and wonder if you would recommend system like
- server: SSG-221E-NE324R Supermicro Storage SuperServer
- cpu: 2 x Intel GOLD 6548Y+ (core/threads 32/64 per cpu)
- memory: 2 TB DDR5
- storage: 24 x 30.72 TB NVMe SSD PCIe 4.0 x4 (<0.5 DWPD) E3.S 7.5mm - Solidigm SSD D5-P5336
- zfs filesystem under datastore (3 x RAIDZ2 (8 drives each))
- network adapter: Mellanox Dual 100Gb/s Ethernet Adapter ConnectX®-6 Dx EN (2x QSFP56), OCP 3.0
I am a little bit worried if it is too much into one chassis, on the other hand it seems also resonable
. it has lots of storage divices directily connected to cpu lines i.e. without extra adapters
. solidigm ssd d5-p5336 are not maybe fastest nvme devices and they are pcie 4.0 etc but still this is nvme (spec says then read 7000 MBait/s and write 3000 MBait/s)
. i dont expect to have very-very busy backuping going on i.e. i dont expect to run info disk dwpd limit) - for say ceph i would maybe consider more performant drives write-wise)
. 30 TB is big maybe in sense if they fail then replacement takes about 3 hours to copy (considering speed on 3 GByte/s and there is 30 000 GByte to go)
. 30 TB vs 15 TB seems big change but cost where i looked at it is not linear i.e. it is only fragment higher than the other (also disk performance is not so good)
. it has nice compute resource in terms of cpu and memory
. i understand general recommendation for zfs is 2-4 G memory per 1 TB of usable data: 24 x 30 is 760 raw, say 1/4 is lost because of redudancy so 540 TB remains as usable: and it is about 4 * 540 equals about 2 TB of memory
. network is fast (connected to Mellanox sn2100 switch which has splittable 100 Gbit/s ports)
. i expect zfs trimming to happen also which i think generally is a good thing
. unfortunately this server is not listed at https://ubuntu.com/certified/servers?q=supermicro&category=Server
. unfortunately for this server is ubuntu not listed at https://www.supermicro.com/en/suppo...ration=X13&form_factor=2U&category=Storage+DP
Also, i looked into other manufacturers offers, Lenovo seems to have SR650 for example, and others too but this supermicro seems most interesting. (i found that is profitable to ask chatgpt 'what lenovo computer is comparable to supermicro SSG-221E-NE324R model' etc, right
I would be very thankful if someone would put their insight into this endeavour!
Best regards,
Imre