CEPH Cluster network setup to separete traffic

[shalom.freitas]

I am hoping someone can point me in the right direction on how to setup up different networks for the different traffic, including "public" and "cluster". I am new to linux / proxmox, so detailed instructions are appreciated.

As of now all 5 servers are running the latest PROXMOX and they are setup in a cluster. They are setup with two network cards in bonding mode for redundancy on 10Gib switch. The server are setup with the IP/subnet 192.168.10.0/24. Here is the setting on one of the servers:

auto lo
iface lo inet loopback

auto eno12399np0
iface eno12399np0 inet manual

auto eno12409np1
iface eno12409np1 inet manual

auto vmbr0v10
iface vmbr0v10 inet static
address 192.168.10.16/24
gateway 192.168.10.1
bridge_ports bond0.10
bridge_stp off

iface eno8303 inet manual

iface eno8403 inet manual

iface eno12419np2 inet manual

iface eno12429np3 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno12399np0 eno12409np1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

iface bond0.10 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0

 

[alexskysilk]

I am hoping someone can point me in the right direction on how to setup up different networks for the different traffic, including "public" and "cluster".

describe which nic(s) is connected, and what your vlans are.

They are setup with two network cards in bonding mode for redundancy on 10Gib switch.

there's almost no point of doing this with only one switch. nics are an unlikely failure point by themselves, but switches are.

 

[shalom.freitas]

describe which nic(s) is connected, and what your vlans are.

there's almost no point of doing this with only one switch. nics are an unlikely failure point by themselves, but switches are.

NICs: eno12399np0 eno12409np1
VLAN 10

That's a good point alexskysilk, I can add another switch to improve that. Thanks!

 

[alexskysilk]

Ok, but you asked to have seperate public and private networks. so in practice, you will need the following networks:

v0- internet/vm traffic
v1- corosync
v2- (optional) corosync ring 2
v3- ceph private
v4- ceph public (may be a bridge if you intend to use it for vm mounted cephfs)

you CAN map all of these to the same pair of nics, but I would advise not to comingle corosync, and ceph performance can be affected comingled.

 

[shalom.freitas]

Ok, but you asked to have seperate public and private networks. so in practice, you will need the following networks:

v0- internet/vm traffic
v1- corosync
v2- (optional) corosync ring 2
v3- ceph private
v4- ceph public (may be a bridge if you intend to use it for vm mounted cephfs)

you CAN map all of these to the same pair of nics, but I would advise not to comingle corosync, and ceph performance can be affected comingled.

So v0 through v4 (these are the vlans, right?) would be a new bridges, all connected to bond0?

how do I specify that corosync uses a different nick than ceph?

 

[alexskysilk]

So v0 through v4 (these are the vlans, right?)

yes.

would be a new bridges

only if you want to use them for vm traffic. normally no- normal interfaces.

all connected to bond0?

they CAN be, but shouldn't.

how do I specify that corosync uses a different nick than ceph?

see https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network

nics and vlan selection are completely in your control to define, which you should do before ever creating a cluster.