Ceph on a VLAN

TwiX

Renowned Member
Feb 3, 2015
310
21
83
Hi, I usually use 2 x 2x10Gb network cards like this :

vmbr0 : LACP bond0 with 2x10Gb ports
ceph (pub/priv) : LACP bond1 with 2x10Gb ports

So we may say that there are 2 dedicated interfaces, one for vms, the other for ceph.
But it requires 4 ports on my switch.

I'm wondering if it could be possible to use Ceph on a VLAN, using a 2x25 Gb card instead (so 50Gb with ceph vlan versus 40 Gb). All services running on a 2x25Gb lacp link.
That's what Dell vxrail do (vlan for vsan, vlan for vmotion, vlans for vms,...)

What do you think ?

Thanks !

Antoine
 
Unless you can make sure with QOS that certain VLANs or packet types have a guaranteed bandwidth / latency you will sooner or later run into problems.

The big issue is if Corosync is on the same physical network and if you have HA guests in the cluster. Worst case scenario is that Ceph is using all the bandwidth which means that the latency is going up for other services. If a node is losing contact to the cluster because the latency for corosync packets is too high, and if you have HA enabled guests on it, the watchdog will become active and after 2 minutes without the cluster the node will fence itself (hard reset).

When used in such a setup and such a situation occurs, it usually affects all nodes and thus it appears as if the whole cluster restarted unexpectedly.
 
  • Like
Reactions: Alwin

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!