VMs and Hosts on same network

[davelong]

Hi All

I've been reading through threads and googling and cannot seem to get this configuration right so was hoping someone could point me to what I am missing.

I have 2 host machines on an internal network, and each host has 2 NIC's installed
I have configured the Hosts as follows:

Host 1:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet static
        address 10.10.1.1/24

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.190/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Host 2:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet static
        address 10.10.1.2/24

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.191/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

The cluster uses the 10..10.1.1 and 10.10.1.2 IP's on eno2 to corosync, this is all working perfectly.

With the default configuration above, I was initially not able to reach any other IP on the internal network, from either Host machine, until a static route was added to the host, for example:

 ip route add 192.168.1.191/32 via 192.168.1.1 dev vmbr0

Without this static route, I was still able to reach the host from my laptop, which is on the 192.168.1.0/24 network (windows laptop on IP 192.168.1.9).

Now, I have a number of VM's on each host. Each VM is configured with it's own Static IP (mostly Ubuntu VM's using netplan) within the same subnet.
eg: 192.168.1.127/24

All Machines are configured to use the same gateway, 192.168.1.1, which is a physical router

ALL VM's that are hosted on a single Host can communnicate with each other, for example:
- VM101 and VM102 on HOST 1 can ping each others IP's and it's own Host IP
- VM103 and VM104 on HOST 2 can ping each other's IP's and it's own Host IP

From my laptop, I can reach ALL VM's on both hosts and both Hosts themselves.

However, I cannot get VM's on Host 1 to Reach VM's on Host 2 and vice versa, and VM's on Host 2 cannot ping Host 1 and VMs on Host 1 cannot ping Host , for example:
- VM101 ping VM103 returns "Destination Host Unreachable"
- VM101 ping HOST 2 returns "Destination Host Unreachable"
etc..

This is despite the netplan config within each VM specifying a default route via the gateway:

    routes:
    - to: default
      via: 192.168.1.1

I have read all kinds of threads, but most are configuring the VM's on a Different Subnet to the Host Network..

My use case is for an internal development network, running Kubernetes and other 3rd party services (DB servers, Message Queing etc etc.) and I am aiming to have all machines on the internal network have unrestricted access to each other.

I would really appreciate any guidance that can be provided as I've tried so many different options now (moving gateway to eno1, moving gateway and IP to eno1, setting up second "bridge" with alternate IP etc etc.) and cannot get the VM's to communicate outside of their host.

Now, I could go and setup a static route on each VM to route to each of the IP's on the other networks via the defaulkt gateway as follows:

 ip route add 192.168.1.101/32 via 192.168.1.1 dev ens118

*using non persistent examples for now but aware would need to persist these...
However, this complicates automated container deployments via build pipelines and before I go down the road of having to manage all the static routes on each VM / Container I would really like to try solve this issue so that ANY VM on ANY HOST, all on the same /24 subnet can all see each other and communicate.

 

[supermicro_server]

Are you using VLAN in your network? I noticed that "vlan aware" tag is enable on vmbr0.
If you don't configure trunk on your phisical switch properly, you won't be able to comunicate from between VMs HOST1 and VMs HOST2.

MM

 

[davelong]

Thanks - We have tried with that switched on an off, and also the below configuration with the vlan-aware tag removed and ip_forward and proxy_arp enabled on eno1

auto lo
iface lo inet loopback

iface eno1 inet manual
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp

auto eno2
iface eno2 inet static
        address 10.10.1.2/24

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.191/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vids 2-4094
        post-up ip route add 192.168.1.0/24 via 192.168.1.1 dev vmbr0

Additionally I have tried a seperate configuration moving the gateway to eno1 and giving eno1 a static IP as follows:

auto lo
iface lo inet loopback

iface eno1 inet static
        address 192.168.1.181/24
        gateway 192.168.1.1
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp

auto eno2
iface eno2 inet static
        address 10.10.1.2/24

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.191/24
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vids 2-4094
        post-up ip route add 192.168.1.0/24 via 192.168.1.1 dev vmbr0

 

[cwakefield]

What I believe is happening is....

The default operation of hosts within a common subnet is they will only use the defined gateway if they need to communicate outside their own subnet. So, a host on 192.168.1.0/24 that needs to communicate with the Internet or another address on 192.168.2.0/24 for example.

So

• vm's on host #1 will broadcast on vmbr0 (their local switch)
  • and will be able to reach another vm on host #1
  • will not be able to reach another vm on host #2
• vm's on host #2 will broadcast on vmbr0 (their local switch)
  • and will be able to reach another vm on host #2
  • will not be able to reach another vm on host #1
• hosts such as your laptop will broadcast to your physical network
  • your proxmox host are connected to that same network and hence see those packets

This explains why your laptop can connect to all vm's regardless of host, but vm's on each on each host can see each other but not see vm's on the other host and vice versa.

When you give the vm a more specific route you are forcing traffic to not be broadcast on vmbr0 but instead be sent to your router which can correctly direct the traffic.

Still new to proxmox myself so not sure if there is a standard approach to solving this.

I'm in VMWare this is solved through a distributed switch it looks like work is being done to get this into proxmox too - https://pve.proxmox.com/pve-docs/chapter-pvesdn.html

 

[davelong]

Thank you - I think you are right.

We're exploring all the options as it looks like we're either going to need to use that experimental feature, or setup a separate VLAN for the VM's.

Once we have a solution I will update here on the outcome.

 

[davelong]

We found the issue and it's now fixed... I (very sheepishly) have to admit it's almost as bad as a "cable wasn't plugged in" type issue - it was the switch the hosts are plugged into had port isolation enabled. Disabled it and everything is working.
Thanks for the assistance and apologies for wasting your time!

 

[supermicro_server]

We found the issue and it's now fixed... I (very sheepishly) have to admit it's almost as bad as a "cable wasn't plugged in" type issue - it was the switch the hosts are plugged into had port isolation enabled. Disabled it and everything is working.
Thanks for the assistance and apologies for wasting your time!

I was sure at 99% about a switch problem
Happy hear you fixed it