Something about ceph and vmbrx is not clear for me

[jpv31]

Hello everyone.

There is something I don't quite understand about CEPH and vmbrx bridges.

When installing CEPH, we are asked to choose an interface, and to create a pool.
In these conditions I chose vmbr1, and I created the pool POOL1.

Now when creating a virtual machine, we are asked for the name of the resource pool and the name of the network interface.

I don't understand why, if I choose the resource pool POOL1, why I am offered all the bridge possibilities (vmbr1, vmbr2, vmbr3....).
I would expect when choosing POOL1 to have vmbr1 as the only choice.

Also, what happens if I create a VM in POOL1, but with bridge vmbr0?
Ceph will not work in this case?

Thank you in advance for your explanations!

 

[Azunai333]

Those are for two different things:

When installing CEPH, we are asked to choose an interface, and to create a pool.
In these conditions I chose vmbr1, and I created the pool POOL1.

Ceph needs a network interface to communicate with the other hosts and Ceph services. This is just for Ceph.
VMs can't access this bridge/network directly. The VM just writes to (or reads from) the virtual HDD and this will go through the defined interface.
See [0] for further details.

why I am offered all the bridge possibilities (vmbr1, vmbr2, vmbr3....).

This defines the bridge to which the VM is going to be connected to. The VM can access the assigned bridge/network.

[0] https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster#pve_ceph_install_wizard

 

[alexskysilk]

Ceph needs a network interface to communicate with the other hosts and Ceph services. This is just for Ceph.
VMs can't access this bridge/network directly.

ceph needs layer 3 communication. it doesn't care if that is over a bridge or not. if OP wanted to, he could make ceph traffic travel over a bridge- but YOU REALLY SHOULDNT.

There are use cases where you would want to offer access to the ceph public interfaces (eg, for cephfs) but even then you would want to separate your ceph public and private interfaces, and keep the private interfaces on its own dedicated interface.

 

[jpv31]

Hello.
Thank you Azunai333 and Alexskysilk pour your answers !

If I understood correctly, it is good to have a network reserved only for CEPH.
Is this network used when we migrate VMs from one server to another ?

Azunai333​

 

[fabian]

the migration network and the ceph network are two separate things. although you can of course configure the same networks to be used for both, if you don't have to, then it's better to keep them separate.

edit: also, a ceph pool and a PVE resource pool are two completely independent things that just happen to share a similar name

 

[jpv31]

Thank for your answer !

I train myself on proxmox and I ask myself lots of questions.

Have a good weekend !