[SOLVED] Weird bridge network issue

[junky]

Hi,

I have a problem I've been trying to solve, but I'm totally stuck. Simplified, I have the next situation: Two proxmox servers in a cluster, one with two vm and one with one vm, and my main desktop pc on the same network. VM Networking devices is bridged (default configuration for the nodes and manual ipv4 for vm). VM's are running Ubuntu Server 20.04. On each VM there is a service I can curl. Each VM basically is a clone with the hostname and ip changed.

This is the setup:
Proxmox_A (192.168.100.103)

- VM_1 (192.168.100.21)​

- VM_2 (192.168.100.22)​

Proxmox_B (192.168.100.104)

- VM_3 (192.168.100.23)​

Desktop_PC (192.168.100.10)

Now the problem, from the desktop I can curl the three VM's no problem. From Proxmox_A I can curl VM_2 and VM_3 no problems and VM_1 it takes like 1 minute to resolve. From Proxmox_B I can curl no problem VM_1 and VM_2 but I takes like 1 minute for VM_3. I'm complete stuck at here, I have no idea how to debug this. I'm guessing is a problem from comunication inside the bridge interface? I have proxmox 7.0-11 runing on two hp proliant gen 8 servers.

This was a bit simplified because this VM are for a kubernetes cluster and are runing several containers. But basically the problem reduces to this.

Any light on this before I become mad will be very appreciated

Some extra notes:
- I can ping all ip's from all machines.
- Port I'm curling: 31416
- Other network services also have trouble, but curl is what I found that I can test.
- I think (but not complete sure) I also have this problem with other VM in the cluster.

 

[junky]

Ok, so I did some progress. If I change VirtIo interface to intel e1000 or Realtek RTL8139 and then everything works. The problem is that I need the speed of VirtIO drivers for intra node communication. If I switch back to VirtIO, the problem persist on the same VM's.

 

[floh8]

do the vms have different MAC-Adress?

 

[junky]

Yes, all VM's have different mac addresses.

It seems it happens with VirtIO network guest drivers on the same Proxmox node. From outside (desktop and the other node) communication seems fine and with e1000 or Realtiek RTL8139 drivers is fine from anywhere (also the same node).

Edit:
I also tried to change mac address of the problematic VM's, same issue persist.

 

[floh8]

i think u use the standard bridge? so try it with the ovs bridge

 

[Tmanok]

Hi There,

What other links do your nodes have? E.g. are they shared with other services such as Corosync (cluster network) or a file server link?

Also, have you tested the speeds of the E1000 to confirm that there really is too much of a performance drop?

Tmanok

 

[junky]

I did change to ovs bridge (and some kubernetes fixing) and now is working!

Thanks!