[SOLVED] Private LAN between servers (3 Node Ceph)

[jsterr]

I want to have a private lan (vmbridge1) that only exists on the hosts and have no additional cabling or dedicated phys. ports. Is this possible?

• I create vmbridge1 with no slaves on each node, gave it a ip adress.
• gave some vms a additional nic with vmbridge1
• put on some ip-adress on those vms

I can only ping between vms that are on the same node. How can I fix that? Can I send this traffic over some already used network ports? Hows the best practice for that? Its traffic that does not need to be routed outside of the switch.

Also seperation via vlan would be nice, because I want to deploy a dhcp server that provides ips only for vms that are on my three nodes. it should not interfer with the local-lan outside thats also available outside of proxmox -ve (vmbridge0)

 

[aaron]

Well, the vmbr interfaces are like switches. If you want them to connect to other nodes, you will have to add some physical port as slave. I would try something like "enoX.VlanID" as slave port on the vmbr to use a VLAN on the physical port. Haven't tested it, but should work.

 

[jsterr]

Well, the vmbr interfaces are like switches. If you want them to connect to other nodes, you will have to add some physical port as slave. I would try something like "enoX.VlanID" as slave port on the vmbr to use a VLAN on the physical port. Haven't tested it, but should work.

Hello Aaron, thanks! How to do that with a bond and lacp? The bond says it doesnt support vlan devices? I have enp67s0f0 and enp67s0f1 which are both in bond0 which is in vmbridge0. I want to use the same physical interfaces but also with lacp/bondig for the vmbridge1.

I created enp67s0f0.33 and enp67s0f1.33 as Linux VLAN but cant add them to a new bond2.

Edit: oh I should maybe read some documentation, linux vlan can also be used with bond

So the way to go:

• create Linux VLAN with bond0.33?
• Put bond0.33 on vmbridge1?
• ??

Im missing something its not working, neither between hosts and also not in vms.

 

[aaron]

Hmm, I would have tried it in the config for the vmbr interface and not the bond directly. But yeah, a dedicated VLAN interface and using that as bridge port could also work

 

[jsterr]

It worked, just make sure to have VLAN Trunk on your Switch-Ports if you use multiple VLANs or at least one Tagged VLAN on it.

auto bond0
iface bond0 inet manual
        bond-slaves enp67s0f0 enp67s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
#VM-01

auto bond0.33
iface bond0.33 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#VM-01

auto vmbr1
iface vmbr1 inet static
        address 10.33.33.1/24
        bridge-ports bond0.33
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#internal LAN