[SOLVED] Proxmox 8.2.2: Destination Host Unreachable but local network is fine

[LostOnTheLine]

I just had this happen to a 2nd node. This one while still setting it up, before joining cluster.

I had this happen to a node I had & I tried a bunch of things but eventually ended up migrating everything off it & re-installing. I thought it had happened as a result of an in-place-upgrade to 8.2.2 from 8.1.something. After taking it down I decided to re-start everything with a new setup. I took down both of my newer nodes, leaving just my original one. I plan to get these 2 going then do the same with that one, except when I re-add it it will be on a different machine.

Here's what happened

Seemingly at random, no new config changes, my node started having a problem with getting updates. I went through these steps, checked network settings, looked online for answers but eventually gave up. Now I'm going through this with a new node that I'm still in the process of setting up. It's been about a week since I set it up initially, but I haven't really finished the setup until now. The other one was updated to 8.2.2 for probably about a week, maybe a bit more or less before it had the same problem.

Here's the steps I've gone through, I'm sharing them with the new node because it's in front of me, but I did the same plus much more, unsuccessfully, with the other.
I went into the CLI & tried a ping

root@boxmox:~# ping 1.1.1.1 -c 4
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
From 169.254.108.185 icmp_seq=1 Destination Host Unreachable
From 169.254.108.185 icmp_seq=2 Destination Host Unreachable
From 169.254.108.185 icmp_seq=3 Destination Host Unreachable
From 169.254.108.185 icmp_seq=4 Destination Host Unreachable

--- 1.1.1.1 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3084ms
pipe 4

I'm accessing the CLI via Windows Terminal SSH so I am connected to the network but

• I tried a local Ping as well

root@boxmox:~# ping 192.168.10.1 -c 4
PING 192.168.10.1 (192.168.10.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.557 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.631 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=0.603 ms
64 bytes from 192.168.10.1: icmp_seq=4 ttl=64 time=0.612 ms


--- 192.168.10.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3106ms
rtt min/avg/max/mdev = 0.557/0.600/0.631/0.027 ms

• I went & checked my network settings, compared them with the original node that's working fine, nothing is different
• I tried turning off vLAN Aware but that didn't change anything
• I turned off, rebooted, left off for an hour, no change
• I tried a different network cable, no change

Does anyone know what I'm doing wrong or how to fix this?

 

[Chris]

I just had this happen to a 2nd node. This one while still setting it up, before joining cluster.

I had this happen to a node I had & I tried a bunch of things but eventually ended up migrating everything off it & re-installing. I thought it had happened as a result of an in-place-upgrade to 8.2.2 from 8.1.something. After taking it down I decided to re-start everything with a new setup. I took down both of my newer nodes, leaving just my original one. I plan to get these 2 going then do the same with that one, except when I re-add it it will be on a different machine.

Here's what happened

Seemingly at random, no new config changes, my node started having a problem with getting updates. I went through these steps, checked network settings, looked online for answers but eventually gave up. Now I'm going through this with a new node that I'm still in the process of setting up. It's been about a week since I set it up initially, but I haven't really finished the setup until now. The other one was updated to 8.2.2 for probably about a week, maybe a bit more or less before it had the same problem.

Here's the steps I've gone through, I'm sharing them with the new node because it's in front of me, but I did the same plus much more, unsuccessfully, with the other.
I went into the CLI & tried a ping

root@boxmox:~# ping 1.1.1.1 -c 4
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
From 169.254.108.185 icmp_seq=1 Destination Host Unreachable
From 169.254.108.185 icmp_seq=2 Destination Host Unreachable
From 169.254.108.185 icmp_seq=3 Destination Host Unreachable
From 169.254.108.185 icmp_seq=4 Destination Host Unreachable

--- 1.1.1.1 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3084ms
pipe 4

I'm accessing the CLI via Windows Terminal SSH so I am connected to the network but

• I tried a local Ping as well

root@boxmox:~# ping 192.168.10.1 -c 4
PING 192.168.7.1 (192.168.7.1) 56(84) bytes of data.
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.557 ms
64 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.631 ms
64 bytes from 192.168.10.1: icmp_seq=3 ttl=64 time=0.603 ms
64 bytes from 192.168.10.1: icmp_seq=4 ttl=64 time=0.612 ms


--- 192.168.10.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3106ms
rtt min/avg/max/mdev = 0.557/0.600/0.631/0.027 ms

• I went & checked my network settings, compared them with the original node that's working fine, nothing is different
• I tried turning off vLAN Aware but that didn't change anything
• I turned off, rebooted, left off for an hour, no change
• I tried a different network cable, no change

Does anyone know what I'm doing wrong or how to fix this?

Hi,
you probably can't reach the default gateway. Try to ping the gateway and double check the routing setup. Are there firewall rules which might interfere, either on the Proxmox VE host itself or on the network.

 

[LostOnTheLine]

you probably can't reach the default gateway. Try to ping the gateway and double check the routing setup. Are there firewall rules which might interfere, either on the Proxmox VE host itself or on the network.

No firewall rules unless Proxmox created them inte4rnally.
& the Ping from my check WAS the gateway itself.

as it stands now & in the other issue there are other hosts that have no problems. No router changes were made. The only consistency is that this happened when both were on 8.2.2 & after they had been for a short number of days without issue

 

[RajatSharma]

Hi, Check your DNS and the routing rules on your network, anything working against your config. Have you configured the gateway in a number format or a dns record format like 192.168.0.1 or gateway.local.proxmox if the answer is DNS record it's high chance that your DNS is at fault.
Regards

 

[Chris]

Two observations which might help you narrow down the issue:

root@boxmox:~# ping 192.168.10.1 -c 4 PING 192.168.7.1 (192.168.7.1) 56(84) bytes of data.

When you ping 192.168.10.1, actually 192.168.7.1 is pinged? Do you have different subnets configured or duplicate IP addresses (depending on the subnet mask)? Please share your network configuration cat /etc/network/interfaces.

root@boxmox:~# ping 1.1.1.1 -c 4 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. From 169.254.108.185 icmp_seq=1 Destination Host Unreachable

The ping seems to reach 169.254.108.185 which however seems to not be able to route this packets further. So you might want to check the routes you packets are taking via tools such as mtr, which you can install via apt install mtr-tiny

 

[UdoB]

From 169.254.108.185

How is this configured? "whois" tells me:

Comment:        Computers use addresses starting with "169.254." when they do not have a manually configured address or when they are not told which address to use by a service on the network.  They are commonly called the "link local" addresses.
Comment:        
Comment:        Routers are not allowed to forward packets sent from an IPv4 "link local" address, so they are always used by a directly connected device.

Also: https://en.wikipedia.org/wiki/Apipa

 

[LostOnTheLine]

Have you configured the gateway in a number format or a dns record format like 192.168.0.1 or gateway.local.proxmox if the answer is DNS record it's high chance that your DNS is at fault.

It's numbers

When you ping 192.168.10.1, actually 192.168.7.1 is pinged? Do you have different subnets configured or duplicate IP addresses (depending on the subnet mask)? Please share your network configuration cat /etc/network/interfaces.

I thought I changed all of them. I don't like giving actual information so I changed the values from 7, actual, to 10 (for this post) I just missed one. I also don't actually call my node boxmox...

So you might want to check the routes you packets are taking via tools such as mtr, which you can install via apt install mtr-tiny

I can't install anything as it can't reach the outside network...

The ping seems to reach 169.254.108.185 which however seems to not be able to route this packets further.

That's not what it means, that's an APIPA address, that's common when a device can't get a DHCP address. This is used in a couple ways that I know of.

1. When a device can't get an IP from DHCP it auto assigns itself in the APIPA range.
2. When the device is an emulated device within another device that doesn't need to communicate locally, this happens in smart devices that are connected to the internet but you can't see in the DHCP settings or connected devices list on your router
3. When using IP masking, again for things that don't need to be reachable.

 

[LostOnTheLine]

So I discovered something. (Thanks ChatGPT)

If I check the route I see this:

root@boxmox:~# ip route
0.0.0.0 dev enp0s31f6 scope link
default dev enp0s31f6 scope link
default via 192.168.10.1 dev vmbr0 proto kernel onlink
169.254.0.0/16 dev enp0s31f6 proto kernel scope link src 169.254.172.193
192.168.10.0/24 dev vmbr0 proto kernel scope link src 192.168.10.115

Which shows that the device is getting an APIPA address on enp0s31f6.
When I systemctl restart networking I can ping 1.1.1.1 right after, but in a minute it goes back to not working.

 

[Chris]

So I discovered something. (Thanks ChatGPT)

If I check the route I see this:

root@boxmox:~# ip route
0.0.0.0 dev enp0s31f6 scope link
default dev enp0s31f6 scope link
default via 192.168.10.1 dev vmbr0 proto kernel onlink
169.254.0.0/16 dev enp0s31f6 proto kernel scope link src 169.254.172.193
192.168.7.0/24 dev vmbr0 proto kernel scope link src 192.168.10.115

Which shows that the device is getting an APIPA address on enp0s31f6.
When I systemctl restart networking I can ping 1.1.1.1 right after, but in a minute it goes back to not working.

Please share your network configuration as requested (without modifying the data you got once again 192.168.7.0/24 dev vmbr0 proto kernel scope link src 192.168.10.115). And provide the current active state by ip addr as well as the output of iptables-save.

hat's not what it means, that's an APIPA address, that's common when a device can't get a DHCP address

Yes, that is what @UdoB already stated, something is not configured correctly. You might want to double check you arp neighbors as well with ip neigh

 

[LostOnTheLine]

Please share your network configuration as requested (without modifying the data you got once again 192.168.7.0/24 dev vmbr0 proto kernel scope link src 192.168.10.115). And provide the current active state by ip addr as well as the output of iptables-save.

No. I will always obscure information when sharing online. that information is neither relevant nor essential to share. Had I not been doing things quick & missed some corrections you never would have known & it never would have made a difference.


So it looks like I've figured it out. I'm not sure why it happened after the update to 8.2.2, but I have a very light LXDE with pretty much everything uninstalled on my Proxmox hosts. It's mostly there so I can have the screens used to display control panels with webscreensaver. For some reason, after the update, but not immediately, & after the initial install, but, again, a couple days later, after a few reboots, it appears connman from LXDE, is trying to force the DHCP check, failing & assigning the APIPA. I did a apt remove connman* & it seems to be working again.
I had this setup on my hosts well before I updated to 8.2.2 & it was never a problem. In fact It's still installed on my node running 7 that I haven't upgraded without any issue. But it seems to be a problem with something between the 2 of them.

When I discovered that the Ping worked for about a minute before failing after a network reset it was clear something was affecting the network, so I looked through the installed packages, removing extra stuff I missed before hoping something would be the cause, when I discovered the connman, conning my network into trying to DHCP.