(Hardware) - Upgrade Proxmox Ceph network from 1G to 10G

[Jota V.]

Hi, we have three servers, running Proxmox 6.

The three nodes are identical.

• 2 x 24 x Intel(R) Xeon(R) CPU E5645
• More than 96 GB RAM per node.
• 1 x Samsung SSD 860 EVO 250GB (Proxmox installation)
• 1 x NVME Samsung SSD 970 EVO 250GB (4 x 48 GB for DB/WALL)
• 4 x 2 TB 7200 rpm Western Digital Drives OSDs

We are running CEPH Cluster using 1G connection. Performance is not very good.

We want upgrade CEPH network 10G with this.

1 x Cisco Nexus N3K-C3064PQ-10GE - https://www.cisco.com/c/en/us/produ...00-series-switches/data_sheet_c78-651097.html
3 x Intel X520-DA2 10G Dual Port SFP+ PCI-E 2.0 x8 Server Adapter
3 x Passive Direct Attach Copper Twinax Cable - https://www.fs.com/products/40109.html

Will this work ? Any changes ?

Actual benchs.

rados bench -p ceph-fs 120 write --no-cleanup

Total time run:         120.518
Total writes made:      2433
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     80.7514
Stddev Bandwidth:       10.4654
Max bandwidth (MB/sec): 104
Min bandwidth (MB/sec): 44
Average IOPS:           20
Stddev IOPS:            2.61635
Max IOPS:               26
Min IOPS:               11
Average Latency(s):     0.792256
Stddev Latency(s):      0.485577
Max latency(s):         3.989
Min latency(s):         0.180162

 

[aaron]

If you do not plan to add more nodes and keep it a 3 Node cluster you can skip the switch, get 3 more DAC cabled and build it as a full mesh Ceph cluster: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

 

[Jota V.]

Thanks aaron, switch is necessary, we will use it for other network equipment and keeps network config more easy. no?

 

[aaron]

Okay, if you need the switch anyway. And yes, the config is simpler and adding another node to your cluster is easier with this too.

 

[Jota V.]

but, will this hardware work together well? i've never used 10G and SFP and DAC cables.

 

[aaron]

Not sure, are there compatibility lists from Cisco?

Other people here in the forum probably have more experience in this regard than I have

 

[multitel]

Jota, if this is still something you are looking to do:

- We have been using the N3K and also a Nexus 5010 switch, with Solarflare cards and amphenol cables.

- Amphenol cables: https://www.ebay.com/itm/NEW-Amphen...e=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

- These also worked with no issues: https://www.ebay.com/itm/LOT-OF-8-C...e=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

- And these were the network cards:
https://www.ebay.com/itm/SFN5322F-S...e=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

- Alternatively, depending on how many servers you will end up with in the future, keep in mind that the N3K has 4 QSFP+ 40Gb ports. You could use these with Melanox 40Gb cards.