Problem with network between quest in cluster

[hermelin]

Hello
I installed Proxmox cluster - like https://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster
All works great.

But I have problem with network comunication between guests on another node (on same node is all ok)

I am using bridged interface

[COLOR=#000000][FONT=monospace]auto vmbr0
[/FONT][/COLOR]iface vmbr0 inet static
        address x.x.x.x
        netmask 255.255.255.0
        gateway x.x.x.x
        bridge_ports eth0
        bridge_stp off [COLOR=#000000][FONT=monospace]        
     bridge_fd 0
[/FONT][/COLOR]

ping node1 -- node2 => OK
ping guest(node1) -- guest(node1) => OK
ping guest(node1) -- guest(node2) => failed
ping node1 --- guest(node2) => failed

I testing disable firewall, tune sysctl like rc_filter, apr_proxy etc.
After tune with tcpdump look like ARP reply doesnt received.

If I ping (firewall disabled)
on local:
21:26:13.398678 ARP, Request who-has .....
21:26:13.638989 ARP, Request who-has.....
.......

on remote:
22:26:01.373251 ARP, Request who-has ...
22:26:01.373354 ARP, Reply ....
22:26:02.373204 ARP, Request who-has...
22:26:02.373291 ARP, Reply .....

No ICMP packet received.

 

[Mr.Holmes]

Hello hermelin,

do you have only one LAN connecting all VMs and nodes in the cluster, connected by eth0 resp. vmbr0?

If so, traffic between guest(node1) and guest(node2) is as follows:


eth0(guest-node1) ---- vethxx(node1) ---- vmbr0(node1) ---- eth0(node1) ----- eth0(node2) ---- vmbr0(node2) ---- vethxx(node2) ---- eth0(guest-node2)

Tracing with tcpdump all these interfaces should show where the ARP reply is lost.

Sometimes I saw similar problems with LINUX bridges (therefore I changed to OVS), mainly when using VLAN.

Also in case of direct IPs for containers (venet0) I saw such problems (therefore I prefer virtual NICs also for containters).

kind regards

Mr.Holmes

 

[hermelin]

Hello,
it look like problem in my provider - each node is connected to the another switch - and problem is probably in STP
if i do route rule - nexthop - directly to gw - all works ok. But if guests must communicate through switch on ARP base is problem.
But very interesting is that physical server working, ping from guest to IPMI modul on another node working, only my bridged device is not working and probably STP is applicated on it. May be that has more MAC addresses.

Can help me if I enable STP on my linux bridges (but in Proxmox howtos is alway disabled) ?

Thanks

 

[Mr.Holmes]

Hello hermelin

it look like problem in my provider - each node is connected to the another switch - and problem is probably in STP

it seems very probable that the problem is in the provider network - it blocks somehow the ARP response. To ensure this assumption tcpdump in both physical eth0 NICs would help.

Obviously provider´s switches have restriction in accepting certain MAC-addresses respectively in handling of the related ARP messages.

Probably related to STP (in the providers network) but I don´t think activation of STP in LINUX bridges would help - why? Here my theory (but really not more than this - can be wrong and a quick experiment activating STP cannot be a mistake):

The switch closest to physical NICs of your server(s) accept only one MAC-address on its port (connected to your server) where to send packets - and this will be the first one it detects, in your case the vmbr0(=eth0) address - the MAC address of the virtual NIC in your VM is more or less ignored. A small test can confirm/infirm this: put a simple (physical) switch between your server and the provider´s connector, connect to that switch another computer and observe if you have the same phenomenon (that only one of them works properly, depending which you connect first). Maybe you can contact your provider and figure out what really happens there.

Kind regards

Mr.Holmes

 

[hermelin]

Hello Holmes,
I solve it by adding special route to all pc (hosts and guests). All traffic routing throught gw of my provider and all works OK.

Thanks for help