Web Gui communication failure (0) between nodes for status panel

[SilverXXX]

Hello,
i'm a new user so sorry if this is a simple question.
I've got a three node cluster, 1 with 4 nic in bonding and 2 with 2 nic in bonding, all with jumbo frames.
I use them for some VM for testing (both common linux vm, some windows, and some linux vm for kubernetes).
Everything was fine since 2 days ago, when i got, when accessing some panels (es Summary -> status) and using some commands from eg node1 to node2 the following errors:
first communication failure (0)
and sometime Connection timed out (596)
Shell works fine thou

Where can i find more logs to understands those errors?
Has anyone got those errors?

 

[Stefan_R]

The GUI accesses the API on the other nodes directly (well, proxied via the first, but directly as in the request is passed directly to node 2). Can you access your nodes without error if you connect straight to them? (e.g. https://node2:8006/)

Have you checked firewall rules that might stop API access between your nodes?

 

[SilverXXX]

yes, when i access a node directly i can see them without troubles, both by ip and dns (it's a small homelab, the router is a fritzbox 7590)
Cluster was created using ip adresses

 

[Stefan_R]

You can take a look at the pveproxy log at /var/log/pveproxy/access.log. (on the node you are initially connecting to)

If that doesn't help, are you sure your bonding is correctly set up?
Maybe try creating an SSH tunnel over one of your nodes to simulate inter-node network traffic without involving Proxmox tools, to make sure your network is fine (e.g. ssh -L 5000:node1_ip:8006 user@node2_ip and then access https://localhost:5000/ to see node1's web interface, if everything is working).

 

[SilverXXX]

This is what i find inside access.log

192.168.178.22 - root@pam [11/06/2019:17:35:42 +0200] "GET /api2/json/nodes/pvesecond/rrddata?timeframe=hour&cf=AVERAGE HTTP/1.1" 200 4420
192.168.178.22 - root@pam [11/06/2019:17:36:10 +0200] "GET /api2/json/nodes/pvesecond/status HTTP/1.1" 500 -
192.168.178.22 - root@pam [11/06/2019:17:36:12 +0200] "GET /api2/json/nodes/pvesecond/status HTTP/1.1" 500 -
192.168.178.22 - root@pam [11/06/2019:17:36:13 +0200] "GET /api2/json/nodes/pvesecond/rrddata?timeframe=hour&cf=AVERAGE HTTP/1.1" 200 4412
192.168.178.22 - root@pam [11/06/2019:17:36:46 +0200] "GET /api2/json/nodes/pvesecond/storage/local/status HTTP/1.1" 596 -
192.168.178.22 - root@pam [11/06/2019:17:36:46 +0200] "GET /api2/json/nodes/pvesecond/storage/local/rrddata?timeframe=hour&cf=AVERAGE HTTP/1.1" 596 -

this is how i setup interfaces

#/etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad

auto vmbr0
iface vmbr0 inet static
        address  192.168.178.111
        netmask  255.255.255.0
        gateway  192.168.178.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        pre-up ip link set eno1 mtu 9000 && ip link set eno2 mtu 9000 && ip link set bond0 mtu 9000

everything is the same for all hosts, it was working just a few days ago. A vm inside host one can access host two.

Tried tunneling and it doesnt work, but ethtool output seems ok...

 

[SilverXXX]

Done some testing, taking out mtu 9000 solved it.
Did i set it in a wrong way? Reading some posts, that seemed the right way

 

[MertsA]

Don't enable jumbo frames, your router doesn't support them.

https://en.avm.de/service/fritzbox/...ow/341_Oversized-packet-support-jumbo-frames/

 

[SilverXXX]

Yes i know, but it did give me troubles even from a notebook directly connected to the switch, or when using ssh from inside the host

 

[MertsA]

Do you have a separate switch or are you referring to the switch built into the router? This still sounds like the expected behavior for setting the wrong MTU on an interface to a switch that doesn't support jumbo frames. If you have a separate switch, why do you think it supports jumbo frames?

 

[SilverXXX]

I have a separate netgear gs748T, all hosts are connected with lacp bonding and 9216 frame size (max possible).
only my work pc reach them thru my fritz router