[SOLVED] 3 nodes stop pinging gateway...

[andrea68]

Hi,

I have a 4 node little cluster on mini PC's in my house.
Nothing fancy: 2 nic (one on USB adapter) on each host.
Physical nic's are dedicated to connect inside SSD storage with 3 brick for storage network.
USB<->Nic (1gb) for management and proxmox cluster.
All work well but after a while 3 nodes don't ping gateway anymore.
A 4th node just added to cluster (almost identical configuration) instead ping the gateway with no problem.
The 3 nodes are reachable from other lan PC's and seems responding as well, cluster work and VM's reach gateway on vmbr0 without issues.
BUT still... if I connect on shell on one of the 3 nodes I cannot upgrade, or check internet 'cause I cannot ping gateway...
This drives me insane really.
I check network hosts configuration multiple times but I cannot explain this behaviour.
No firewall, no rules on hosts etc...
I try to restart the physical switch and the whole cluster without any change...

This is one of the proxmox host that can't go on gateway (192.168.1.1)

--- ---- --- ---
auto lo
iface lo inet loopback

iface enxa0cec872d15c inet manual

auto enp1s0
iface enp1s0 inet static
address 10.10.10.30/24
#stor03

auto vmbr0
iface vmbr0 inet static
address 192.168.1.53/24
gateway 192.168.1.1
bridge-ports enxa0cec872d15c
bridge-stp off
bridge-fd 0

iface wlp2s0 inet manual
--- ---- --- ---

This it the 4th host that instead CAN ping gateway:

--- ---- --- ---
auto lo
iface lo inet loopback

iface enx000ec6bf0af9 inet manual

auto enp0s25
iface enp0s25 inet static
address 10.10.10.40/24
#stor04

iface wlp2s0 inet manual

iface enx00e04c6c3fb4 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.54/24
gateway 192.168.1.1
bridge-ports enx00e04c6c3fb4
bridge-stp off
bridge-fd 0
--- ---- --- ---

Can anyone have any idea of what happens here?

 

[shrdlicka]

Hi,

can the nodes reach anything else in the 192.168.1.0/24 network. They can ping each other over the 10.10.10.0/24 network? How does ip route look like for the working and non working node?

Is the firewall active on some of the nodes?

 

[andrea68]

No firewall active, nodes ping each others, and other IP on LAN, just don't ping gateway (and outside) anymore...

But fortunately I found the problem: is the gateway itself.
Is an old Telecom modem/router: I disable the DHCP on this modem to activate on a pihole, inside my lan.
This lead to the problem.
If I re-enable the dhcp on the modem/router then proxmox hosts come back to ping and go outside, without issues...
I guess I need to replace this crappy hardware modem/router, nothing wrong in my proxmox configurations...
After spending hours for nothing finally another similar issue found on this forum lead me to the right path