[SOLVED] No connection between VM's in same vLAN on same node

[FrankvdAa]

I've been running PVE 6.4 up till now on a single HP Proliant DL380 Gen8 server for my home setup. This has been working fine ever since, but now I'm busy replacing the HP with a new PVE 7.2 cluster setup based on two Intel NUC units and a RPi as qdevice.

These NUC's are each connected to the switch with a single uplink on which I have configured vlan1 (untagged) and all other vLAN's tagged. This interface is connected to bridge 'vmbr0' in PVE.

All VM's that I have, have one NIC assigned that is mapped to 'vmbr0' and the corresponding vlan, f.e. ...bridge=vmbr0,tag=300

These VM's can communicate with other equipment within the same vlan perfectly fine.

I have one VM (my OPNSense firewall) that I assigned three virtual NIC's to:
- ...bridge=vmbr0,tag=6 (= vtnet0)
- ...bridge=vmbr0,tag=4 (= vtnet1)
- ...bridge=vmbr0 (= vtnet2)

Within OPNSense I have configured 'vtnet0' as WAN, 'vtnet1' as IPTV and 'vtnet2' as LAN. Besides that, I have created several vlan interfaces on 'vtnet2', f.e. vlan 300.

Now comes the issue; I cannot communicate with any VM's through OPNSense that are running on the same PVE node. When I run all my VM's on node1 and run only OPNsense on node2, all is working fine. I'm not using the PVE firewall on any of the virtual NIC's.

It looks like PVE is not routing/forwarding the packets between vmbr0 (tagged) and vmbr0v300. The option 'VLAN aware' is set disabled, but even with it enabled it is not working.

As I said before; I have similar setup on PVE 6.4 that is working fine. I found below article that looks a bit like my issue, but I'm not using firewall on any of the virtual NIC's:
https://forum.proxmox.com/threads/p...irewall-for-vms-only-on-the-same-node.108485/

Please advise.

 

[FrankvdAa]

To be complete; on the HP I'm using OpenvSwitch Bridge in stead of Linux Bridge. Therefore I've also tested on the PVE 7.2 cluster with OpenvSwitch, but that didn't make a difference. If OpenvSwitch is required for this setup, than that's what it needs to be, but otherwise I prefer to keep it simple and use Linux Bridge.

 

[jenssen99]

Hello Frank, same issue here, I made a workaround like you described to move the OPNSense VM to a different host, that worked. Looking forward to a solution to run all VM's on one host.

 

[vesalius]

Please post your /etc/network/interfaces file.

 

[FrankvdAa]

After some more reading last night, I understood that OVS should be doing just fine on this matter, so I gave it another try and now it seems to work fine.

Maybe I didn't give it a reboot the first time I tried OVS. Could that have been the reason for it not to work properly?

 

[FrankvdAa]

This is current (working) interface configuration:

auto lo
iface lo inet loopback

auto enp89s0
iface enp89s0 inet static
address 10.0.0.1/30
mtu 9000

iface wlo1 inet manual

auto enp88s0
iface enp88s0 inet manual
ovs_type OVSPort
ovs_bridge vmbr0

auto management
iface management inet static
address 172.16.2.251/24
gateway 172.16.2.254
ovs_type OVSIntPort
ovs_bridge vmbr0

auto vmbr0
iface vmbr0 inet manual
ovs_type OVSBridge
ovs_ports enp88s0 management

I will see if I can revert the configuration on one node and have the same issue again.

 

[FrankvdAa]

Just reverted the OVS configuration and configured Linux Bridge; now the issue is back again.

Current interface configuration:

auto lo
iface lo inet loopback

auto enp89s0
iface enp89s0 inet static
address 10.0.0.1/30
mtu 9000

iface wlo1 inet manual

iface enp88s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 172.16.2.251/24
gateway 172.16.2.254
bridge-ports enp88s0
bridge-stp off
bridge-fd 0

Reverting back to OVS without reboot isn't working either, only after reboot it starts working. So I probably forgot to reboot the first time I tried OVS.

 

[vesalius]

Is 172.16.2.0/24 a vlan on opnsense?

to make sure I understand you, you have defined created all the vtnet2 vlans within native OPNsense settings and I assume at the other individual vm/lxc interfaces, but have not specified them at the proxmox OPNsense vm network interface settings? Whereas vtnet0 and vtnet1 vlans (4&6) are defined at the proxmox OPNsense vm network interface level? are vlan4 and vlan6 created/defined within the OPNsense settings?

 

[FrankvdAa]

172.16.2.0/24 is vlan1, which is untagged on the physical port and thus on vmbr0. It is the PVID on the switch port.

Except for vlan4 and vlan6, all vlans are created as vlan interfaces within OPNsense on vtnet2.

The other VM's have one vnic on vmbr0 and have their corresponding vlan tagged in Proxmox.

I will share some the Proxmox configs of some VM's later.

Edit: here are three configs...

OPNsense:

root@pve01-node1:~# qm config 100 | grep ^net
net0: virtio=0E:83:CC:8D:4B:29,bridge=vmbr0,queues=2,tag=6
net1: virtio=FE:BC:8E:2A:FB:5E,bridge=vmbr0,queues=2,tag=4
net2: virtio=DE:1B:FD:F4:0F:3E,bridge=vmbr0,queues=2

VM in vlan300:

root@pve01-node1:~# qm config 200 | grep ^net
net0: virtio=00:0c:29:6e:d7:cf,bridge=vmbr0,tag=300

VM in vlan1:

root@pve01-node1:~# qm config 201 | grep ^net
net0: virtio=00:0c:29:78:c12:e4,bridge=vmbr0

 

[vesalius]

from the outside looking in you would need to make vmbr0 vlan aware. For the Linux bridge variant have you tried making vmbr0 vlan aware and then rebooting, since that seems to work some magic with OVS?

I run a similar setup with OPNsense vm over a vlan aware Proxmox Linux bridge.

 

[FrankvdAa]

I did try with setting the Linux Bridge VLAN aware, but not sure anymore whether I rebooted after that. Will run that test and let you know the result.

 

[FrankvdAa]

Just reverted back to Linux Bridge with VLAN aware option enabled:

auto lo
iface lo inet loopback

auto enp89s0
iface enp89s0 inet static
address 10.0.0.1/30
mtu 9000

iface wlo1 inet manual

iface enp88s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 172.16.2.251/24
gateway 172.16.2.254
bridge-ports enp88s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

After rebooting, it seems to work too. So it needs a reboot after enabling 'VLAN aware' option and/or after moving to OVS.

I think we can mark this as solved?!

 

[jenssen99]

Very nice, will check this too, thanks for the efforts.