Best to ask first - Ceph network on 10GB or 1GB if that is all you have?

[damonpillinger]

Hi Guys (and Gals)

I have a 4 node cluster and am going to install CEPH again from scratch.

each node has 1 x 10GB NIC and either 1 or 2 x 1GB NIC

I have (seperate) both 10GB Switch and a 1GB switch.

less than 20 VM's (all VM's are windows 48GB C: drives)
Lots of Ram
will set up the CEPH with Tiering using SSD's (1TB) and then HDD's on each node.


Which is the best NETWORK combination to allow a good CEPH system (my guess is B but best to ask)

A)
Proxmox on 10GB
CEPH Public and Cluster network on 1GB

B)
Proxmox on 1GB
CEPH Public and Cluster network on 10GB

C)
Proxmox and CEPH Public network on 1GB
CEPH Cluster network on 10GB

D)
Proxmox and CEPH Public network on 10GB
CEPH Cluster network on 1GB

E)
Proxmox and CEPH Cluster network on 1GB
CEPH Public network on 10GB

F)
Proxmox and CEPH Cluster network on 10GB
CEPH Public network on 1GB

G)
just BOND everything together on each node

Any other combinations ( but I think I have them all covered)

thanks
Damon

 

[LnxBil]

Any other combinations ( but I think I have them all covered)

Add more NICs or replace the ones already in there is no option?

 

[aaron]

Have CEPH use the 10GBit network, no matter what.
If possible give the Proxmox Cluster traffic (corosync) it's own 1GBit network and use the second 1GBit for the production traffic to and from the VMs.

Corosync can use more than one network as a fallback in case the first fails or is congested: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

 

[damon1]

thanks guys
i will add a few 1GB cards to 2 of the boxes which will give me 1x 10 , 2x1GB in each box then
"
Have CEPH use the 10GBit network, no matter what.
If possible give the Proxmox Cluster traffic (corosync) it's own 1GBit network and use the second 1GBit for the production traffic to and from the VMs.
"

many thanks
again
damon