Is it safe to use same switch for CEPH and CLUSTER networks?

[silvered.dragon]

I'm planning a 7nodes proxmox cluster. Of those 7nodes, 3 will have a ceph shared storage. Each node is equipped with 3x RJ45 and 2x SFP+ network interfaces.
I know that is best to have separated networks for CEPH, PROXMOX CLUSTER and LAN, but I was thinking if is a good Idea to use a setup with VLANS and bonds for CEPH + PROXMOX CLUSTER on the same switch:

• 2 switches like the mikrotik CRS328-24P-4S+RM that consist in 24x RJ45 and 4x SFP+
• each switch with 2 VLANS, one for the entire RJ45 side for the PROXMOX CLUSTER traffic, and one for the entire SFP+ side for CEPH traffic
• I'm going to connect each node interface to the two switches in an active-backup configuration (or even MLAG), one for CEPH (SFP+ )and one for PROXMOX CLUSTER(RJ45)

any downside/suggestion about this?

Moreover I have spare network cards, so for me is easy to replace a faulty one, but is difficoult to replace the entire switch in case of failure due the cost and availability, so I'd like to know if active-backup is more suitable in comparision with MLAG, or there is some kind of performance improvent using MLAG except the tollerance of the faulty interfaces on node side.

many thanks

 

[floh8]

you should add a extra dual 10G Card to the Ceph nodes. So u can built a 4x10GBit LAG for Ceph Cluster, Storage Access, VM Access and optional Live-Migration in separate VLANS.
1x 1GBit RJ45 for Managment, 1x 1GBit RJ45 for Proxmox-Cluster, 1x 1GBit RJ45 for Live-Migration or Internet Traffic to Firewall-VM. Also, u can create it with LAG+VLAN.
Of Course u need a Switch with more 10 GBit Ports. 4x2 Ports + 3x4Ports = 20 Ports. If u like Mikrotik u can buy 2xCRS317-1G-16S+RM + 2xCRS326-24G-2S+RM.
MLAG+LACP is better for bandwith usage.

 

[silvered.dragon]

Sorry why I have to use so many interfaces? Is my below scheme wrong? I'm not interested to have failover on public lan, because I have many replacements for this switch, and this interface is not affecting the health of the cluster, I can replace the public lan switch without loosing anything just a little of maintenance. I think that I can add even another ceph node if I want with this setup..

 

[floh8]

Sorry why I have to use so many interfaces?

Because of the performance...or its not important for you?
Every Node need a STorage network. 1 GBit is to slow. You need bigger Switch, I told you.

Is my below scheme wrong?

mlag is missing. Storage network? Live Migration? Look upwards! Backup network missing.
LACP Active Backup is stupid.

I'm not interested to have failover on public lan

yes, thats not neccesary

I think that I can add even another ceph node if I want with this setup..

no, mlag!!!

 

[silvered.dragon]

Just to understand..

Because of the performance...or its not important for you?

of course, but which is the buttleneck? I have 10K SAS mechanical drives in the ceph cluster, is a 10GB connection not enough? Of course I can improve the public LAN to SFP+, but for the amount of traffic in my company is enough a 1GB link per server..

mlag is missing

Sure I can use mlag and is supported in the mikrotik switches, but I'm not interested in failover for the nics just for the switches, so i was thinking that an active backup bonding setup is enough. If a nic goes down I can easily replace it with spare parts, but if a switch goes down is a huge problem for the health of the entire cluster. So I prefer to preserve one SFP+ port for a future CEPH node and avoid MLAG.

Storage network?

Sorry in my scheme the red marked one, is not a ceph storage network for you?

Live Migration?

The VMS that has critical services, are all in the ceph nodes, the other nodes are just utilities, datas and backupservers. So Live migration will be only on those 3 nodes, In my current setup I use the ceph network for migration, I know that is not a good practice but it always worked superfast.. anyway sure I can add other SFP+ interfaces just for migration, but this doesn't means that my schema is wrong..

 

[alexskysilk]

question- how are nodes 4-7 access storage? are you planning on having ceph public network on the 1gb?

1gb with spinning drives is not going to make for a very, umm, quick solution.

 

[floh8]

Sorry in my scheme the red marked one, is not a ceph storage network for you?

I do not mean the Ceph Cluster Network but the PVE Node to Storage Network.

 

[silvered.dragon]

question- how are nodes 4-7 access storage? are you planning on having ceph public network on the 1gb?

1gb with spinning drives is not going to make for a very, umm, quick solution.

they have local storage , they do not have access to ceph storage, they are nodes just for utilities, like scanner server, nas, backup of workstations and other non critical services. I will live migrate only between 1-3. 4-7 will never migrate, they are all together in one cluster just for unify the management of all my proxmox servers.

I was thinking to not separate ceph cluster network from ceph public network, so they are both on same 10GB interface

 

[floh8]

An important point to know - Don't you think so?

 

[silvered.dragon]

An important point to know - Don't you think so?

Probably I'm missing something here.. just to clarify

• nodes 1-3 are the only nodes that will have access to the ceph storage. ceph cluster network and ceph public network are on the same 10GB connection. So this will be the speed beetween nodes 1-3. I will migrate VMs only between those 3 nodes.
• nodes 4-7 has local storage, no access to ceph network, no live migration. They are in the same cluster just for unify the management of all servers.