Question regarding PVE Network Model with Ceph

[cookiesowns]

Hi,

I'm working on setting up a new LAB environment for PVE, trying to test Ceph + Proxmox's clusters capability for a setup.

We have the following nodes:

7x E3 v2, 32GB RAM, with 2x1G LACP bonded down to a multi chassis switch
4x E5 v1, with 8x400GB SSD with 2x1GB LACP bonded down to multi chassis switch and 4x10GbE for Ceph network
4x E5 v3 with 2x1GB LACP bonded down, 2x10GbE for ceph, and 1x10GbE for VM traffic.

I was looking at the documentation for the network model, but I wasn't able to fully determine if it's possible to really have granular control without resorting to elaborate configurations.

Really what I want is:

4x E5 v3 nodes have the 2x10GbE be dedicated just for ceph storage traffic, these nodes will strictly be acting as rdb clients, not servers. the 2x1GbE lacp will be used for management and also cluster network traffic. The remaining 10GbE will be on 2 different vlans strictly for VM traffic.

7x E3 v2 nodes will have multiple vlan's trunked down. Let's say, Cluster network, management/proxmox network, VM network, and also a way to access the ceph rdb's ( these will be hosting very light VM's )

4x E5 v1's will be dedicated to ceph, so I think network configuration here is a bit easier. 2x1Gb LACP with Cluster network/management, and 4x10GbE strictly in Ceph

I already have 4 subnets allocated for this.

My main question is, is it necessary to create a bridge for each vlan, or do we really only need the bridge for if VM's will live on that vlan?

EDIT: I've just read that by default VM migration will happen over the defined cluster network, is there a way to change this behavior? Say we want this to happen over the single 10GbE interface for VM Traffic?

 

[manu]

> My main question is, is it necessary to create a bridge for each vlan, or do we really only need the bridge for if VM's will live on that vlan?
Not 100% sure if I understand your question, but when you assign a NIC to a VM you create, this NIC will be assigned to a bridge of the proxmox *and* you can specify a VLAN tag. So no need to create a bridge for each VLAN.

> EDIT: I've just read that by default VM migration will happen over the defined cluster network, is there a way to change this behavior? Say we want this to happen over the single 10GbE interface for VM Traffic?
In that case create the cluster over the 10GB network.

I am not sure it would make sense to have a migration network separated from the cluster network, because in case of HA, if your migration network is down our HA engine would not know about it, and could possibly start migrating VMs without knowing it's unappropriate.

 

[cookiesowns]

Hi,

What I mean was, is it necessary to have a bridge active ( vmbr0 & vmbr1 ) for default interface?, but so far it appears Proxmox binds on all interfaces / addresses for Web UI & other items, so it might not be necessary.

So essentially, can we have vmbr0 be on a separate vlan, and assign an address to say bond0/bond1 within another VLAN for management purposes, and only use the bridge for VM traffic on separate vlan?

Per my testing, and base on my guesses, this seems to be a yes, but not sure if it might cause any issues.

As per migration, it makes sense, migration speeds seem to be reasonable so far, haven't tested an HA failover yet, but I figure it shouldn't be too bad.

 

[manu]

Yes PVE binds to all interfaces so you can access the GUI via other interfaces, going through vmbr0 is not mandatory.
Remember you can also create as many bridges as you want on ( vmbr1..10) on the Proxmox host.