[SOLVED] Yet Another Gateway Issue

[spikerguy]

Hello everyone,

As the title suggest I am not able to ping gateway at all. I have search the forum and online for similar issue, put it 2 full days to figure it out on my own and have not found a solution for my case. So here I am on the forum asking for help.

Here are some logs

root@prox:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.100.2/24
gateway 192.168.100.1
bridge-ports enp1s0
bridge-stp off
bridge-fd 0

iface wlp2s0 inet manual

root@prox:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
link/ether 7c:83:34:b0:ee:91 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether a8:93:4a:df:18:3f brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7c:83:34:b0:ee:91 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.2/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::7e83:34ff:feb0:ee91/64 scope link
valid_lft forever preferred_lft forever
root@prox:~# ip route
default via 192.168.100.1 dev vmbr0 proto kernel onlink
192.168.100.0/24 dev vmbr0 proto kernel scope link src 192.168.100.2
root@prox:~# ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
From 192.168.100.2 icmp_seq=1 Destination Host Unreachable
From 192.168.100.2 icmp_seq=2 Destination Host Unreachable
From 192.168.100.2 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.100.1 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4074ms
pipe 4
root@prox:~# timedatectl
Local time: Mon 2024-02-19 13:48:56 +03
Universal time: Mon 2024-02-19 10:48:56 UTC
RTC time: Mon 2024-02-19 10:48:56
Time zone: Asia/Bahrain (+03, +0300)
System clock synchronized: no
NTP service: active
RTC in local TZ: no
root@prox:~# hwclock
2024-02-19 13:48:58.358662+03:00

I have tried almost all the solutions from online.
1. Check hwclock on bios and correct it, sync timedatectl as per current time and then sync it with hwclock too.
2. Check for network issue where I have opnsense router with multiple pc's connected. I can ping the proxmox from my laptop but not from my router nor from my proxmox to my router. I have no idea how is this even possible. The only theory I can come up with is the switch is the key here where prox and laptop is on.
3. I suspect the broadcast IP next but I am not sure if that will make any difference. Please suggest.


I can ping every other pc on the network but not the gateway.

root@prox:~# ping 192.168.100.5
PING 192.168.100.5 (192.168.100.5) 56(84) bytes of data.
64 bytes from 192.168.100.5: icmp_seq=1 ttl=128 time=2.98 ms
64 bytes from 192.168.100.5: icmp_seq=2 ttl=128 time=2.33 ms
^C
--- 192.168.100.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.334/2.655/2.976/0.321 ms
root@prox:~# ping 192.168.100.165
PING 192.168.100.165 (192.168.100.165) 56(84) bytes of data.
64 bytes from 192.168.100.165: icmp_seq=1 ttl=64 time=0.617 ms
64 bytes from 192.168.100.165: icmp_seq=2 ttl=64 time=0.755 ms
^C
--- 192.168.100.165 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1015ms
rtt min/avg/max/mdev = 0.617/0.686/0.755/0.069 ms
root@prox:~# ping 192.168.100.10
PING 192.168.100.10 (192.168.100.10) 56(84) bytes of data.
64 bytes from 192.168.100.10: icmp_seq=1 ttl=255 time=0.497 ms
^C
--- 192.168.100.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.497/0.497/0.497/0.000 ms
root@prox:~# ping 192.168.100.100
PING 192.168.100.100 (192.168.100.100) 56(84) bytes of data.
64 bytes from 192.168.100.100: icmp_seq=1 ttl=64 time=0.663 ms
64 bytes from 192.168.100.100: icmp_seq=2 ttl=64 time=0.297 ms
^C
--- 192.168.100.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1014ms
rtt min/avg/max/mdev = 0.297/0.480/0.663/0.183 ms
root@prox:~# ping 192.168.100.159
PING 192.168.100.159 (192.168.100.159) 56(84) bytes of data.
64 bytes from 192.168.100.159: icmp_seq=1 ttl=64 time=1.57 ms
^C
--- 192.168.100.159 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.565/1.565/1.565/0.000 ms
root@prox:~# ping 192.168.100.165
PING 192.168.100.165 (192.168.100.165) 56(84) bytes of data.
64 bytes from 192.168.100.165: icmp_seq=1 ttl=64 time=0.963 ms
64 bytes from 192.168.100.165: icmp_seq=2 ttl=64 time=0.751 ms
^C
--- 192.168.100.165 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.751/0.857/0.963/0.106 ms
root@prox:~# ping 192.168.100.180
PING 192.168.100.180 (192.168.100.180) 56(84) bytes of data.
64 bytes from 192.168.100.180: icmp_seq=1 ttl=64 time=1.37 ms
64 bytes from 192.168.100.180: icmp_seq=2 ttl=64 time=0.860 ms
^C
--- 192.168.100.180 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.860/1.113/1.366/0.253 ms
root@prox:~# ping 192.168.100.254
PING 192.168.100.254 (192.168.100.254) 56(84) bytes of data.
64 bytes from 192.168.100.254: icmp_seq=1 ttl=64 time=0.803 ms
64 bytes from 192.168.100.254: icmp_seq=2 ttl=64 time=0.784 ms
^C
--- 192.168.100.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1013ms
rtt min/avg/max/mdev = 0.784/0.793/0.803/0.009 ms

root@prox:~# nmap -sP 192.168.100.0/24
Starting Nmap 7.93 ( https://nmap.org ) at 2024-02-19 14:13 +03
Nmap scan report for 192.168.100.5
Host is up (0.0022s latency).
MAC Address: XXXXXXXXXXXXXXXX (Dell)
Nmap scan report for 192.168.100.10
Host is up (0.00040s latency).
MAC Address: XXXXXXXXXXXXXXXX (Hewlett Packard)
Nmap scan report for 192.168.100.100
Host is up (0.00016s latency).
MAC Address: XXXXXXXXXXXXXXXX (Grandstream Networks)
Nmap scan report for 192.168.100.159
Host is up (0.00020s latency).
MAC Address: XXXXXXXXXXXXXXXX(Unknown)
Nmap scan report for 192.168.100.165
Host is up (0.00055s latency).
MAC Address: XXXXXXXXXXXXXXXX (BizLink (Kunshan))
Nmap scan report for 192.168.100.180
Host is up (0.00081s latency).
MAC Address: XXXXXXXXXXXXXXXX(Grandstream Networks)
Nmap scan report for 192.168.100.254
Host is up (0.00014s latency).
MAC Address: XXXXXXXXXXXXXXXX Unknown)
Nmap scan report for 192.168.100.2
Host is up.
Nmap done: 256 IP addresses (8 hosts up) scanned in 27.88 seconds

Can anyone advice where I should look into ? I will try to re-install it again. I have used `proxmox-ve_8.1-2` latest image burned it on usb using balena etcher, this is not the first time I am installing proxmox, but still feels like I am a novice hahaha.

Even Nmap cannot find the gateway in the scan.

As it cannot ping the gateway it is not able to resolve DNS too. I face exact same issue when I create a VM and bridge it to the network.

 

[leesteken]

Maybe the gateway simply does not respond to pings? The network connection of Proxmox appears to be working fine withing your local network. Issues between Proxmox and outside of your local network (the internet) are usually caused by the gateway/router/firewall (which one do you use?) and are outside of Proxmox.

As it cannot ping the gateway it is not able to resolve DNS too. I face exact same issue when I create a VM and bridge it to the network.

Ping (ICMP) and DNS (UDP and/or TCP) are different things and a firewall might block one but allow the other. Are you concerned about ping or DNS? Can you ping the gateway from other systems in your local network? Does DNS resolving work for other systems in the local network? What DNS server did you configure (cat /etc/resolv.conf)?

EDIT: Can you ping 1.1.1.1 or try 1.1.1.1 as DNS server, just for testing?

 

[spikerguy]

Maybe the gateway simply does not respond to pings? The network connection of Proxmox appears to be working fine withing your local network. Issues between Proxmox and outside of your local network (the internet) are usually caused by the gateway/router/firewall (which one do you use?) and are outside of Proxmox.

Ping (ICMP) and DNS (UDP) are different things and a firewall might block one but allow the other. Are you concerned about ping or DNS? Can you ping the gateway from other systems in your local network? Does DNS resolving work for other systems in the local network? What DNS server did you configure (cat /etc/resolv.conf)?

EDIT: Can you ping 1.1.1.1 or try 1.1.1.1 as DNS server, just for testing?

Thank you for the quick response and sorry that I mixed DNS here but as gateway is not reachable there is no DNS. cause gateway is routing the dns to my ISP IP.
Yes I can ping the gateway from other pc's as that's the first level of test I did. That means the router is not blocking pings(ICMP) nor DNS(UDP)

Resolve have the DNS as per the other computers too on prox. I am using a single opnsense router which is the gateway and a switch to connect to all the pc's.

Currently trying to re-install and now it is stuck on detecting country, will wait for 20mins as it is doing the traceroute as commented on some reddit post, else will move to console install.

I suspect that the clock in bios keeps resetting while the date is correct. IDK if this is CMOS Battery issue or something to do with BIOS itself.

 

[leesteken]

Thank you for the quick response and sorry that I mixed DNS here but as gateway is not reachable there is no DNS. cause gateway is routing the dns to my ISP IP.
Yes I can ping the gateway from other pc's as that's the first level of test I did. That means the router is not blocking pings(ICMP) nor DNS(UDP)

Resolve have the DNS as per the other computers too on prox. I am using a single opnsense router which is the gateway and a switch to connect to all the pc's.

Currently trying to re-install and now it is stuck on detecting country, will wait for 20mins as it is doing the traceroute as commented on some reddit post, else will move to console install.

Sound like tour opnSense is blocking the traffic from 192.168.100.2. Maybe you can find out why in the logs on opnSense? Can you please try pinging something outside like 1.1.1.1, to check if other traffic is routed (maybe it only blocks connections to itself)?
Are the other systems using DHCP instead of a static address? Is .2 maybe inside the DHCP range (which is sometimes blocked when the system does not use DHCP)?
Can you reach https://192.168.100.2:8006 from opnSense (using wget or curl or whatever it provides)?

 

[spikerguy]

Sound like tour opnSense is blocking the traffic from 192.168.100.2. Maybe you can find out why in the logs on opnSense? Can you please try pinging something outside like 1.1.1.1, to check if other traffic is routed (maybe it only blocks connections to itself)?
Are the other systems using DHCP instead of a static address? Is .2 maybe inside the DHCP range (which is sometimes blocked when the system does not use DHCP)?
Can you reach https://192.168.100.2:8006 from opnSense (using wget or curl or whatever it provides)?

Thank you for reply once again.

I cannot ping outside the network too, pinging 1.1.1.1 or 8.8.8.8 fails just like pinging gmail.com.
OpnSense is not blocking anything on lan interface.
DHCP range is 150-250 and Prox is static hence outside DHCP range.

Good point let me wget the url from opnsense and see what happens. curl did not respond at all just gets stuck.

Thanks.

UPDATE:
I found something strange BIOS clock keeps going back to 2pm after every shutdown.
but hwclock in proxmox shows current time correctly. Now another question is if it have correct time then why does it not connect to the network. ARP on Opnsense shows 192.168.100.2 is present.

 

[leesteken]

Currently trying to re-install and now it is stuck on detecting country, will wait for 20mins as it is doing the traceroute as commented on some reddit post, else will move to console install.

There was a long thread here about issues with country detection: https://forum.proxmox.com/threads/proxmox-installation-trying-to-detect-country.134301/post-632648

I suspect that the clock in bios keeps resetting while the date is correct. IDK if this is CMOS Battery issue or something to do with BIOS itself.

That should not interfere with your network connection, should it?

I cannot ping outside the network too, pinging 1.1.1.1 or 8.8.8.8 fails just like pinging gmail.com.
OpnSense is not blocking anything on lan interface.
DHCP range is 150-250 and Prox is static hence outside DHCP range.

So no traffic is routed outside but the switch allows traffic within the local subnet. Nothing in the logs of opnSense? Maybe the switch is blocking between Proxmox and opnSense? Maybe try a different IP address instead of .2?

Good point let me wget the url from opnsense and see what happens. curl did not respond at all just gets stuck.

Also sound like traffic between opnSense and Proxmox is not working. What kind of switch is used? Do you maybe have a duplicate MAC or IP in the network?

 

[spikerguy]

There was a long thread here about issues with country detection: https://forum.proxmox.com/threads/proxmox-installation-trying-to-detect-country.134301/post-632648

That should not interfere with your network connection, should it?

So no traffic is routed outside but the switch allows traffic within the local subnet. Nothing in the logs of opnSense? Maybe the switch is blocking between Proxmox and opnSense? Maybe try a different IP address instead of .2?

Also sound like traffic between opnSense and Proxmox is not working. What kind of switch is used? Do you maybe have a duplicate MAC or IP in the network?

Wow you're on fire. Loving proxmox community

Thanks again for the quick response.
I flashed 7.4 booted into live and it got DHCP and when I ping it from opnsense router it works. Now waiting for the installation to complete so I can test if it will work with static ip.

UPDATE: The moment proxmox 7.4 installer rebooted, I lost ping from router to proxmox ip. so not sure whats happening after install. WOW I even kept the same DHCP IP like on live boot and rebooted and it stopped pinging.

 

[leesteken]

I flashed 7.4 booted into live and it got DHCP and when I ping it from opnsense router it works. Now waiting for the installation to complete so I can test if it will work with static ip.

PVE 7.4 uses an older kernel. Maybe there is a known issue with your network controller and Linux kernel 6.5? Try searching the forum for your device.

 

[spikerguy]

PVE 7.4 uses an older kernel. Maybe there is a known issue with your network controller and Linux kernel 6.5? Try searching the forum for your device.

It uses old RTL 8189 driver so I am sure it have nothing to do with the kernel as live boot have the same kernel as the one installed.

I feel some issue on proxmox network stack once installed. I will install ubuntu server on the device just to make sure it works on my network in and outside DHCP range. then will come back to proxmox inside VM on my laptop as host.

Any other suggestion to debug this strange network issue ?

OK I AM BLOWN AWAY NOW.
This is Proxmox 8.1.2 on VM on my laptop host. It can ping other pc but not gateway. so It is definitely not pc problem. Something seriously missing on proxmox. Let me try 6.4.1

 

[leesteken]

It uses old RTL 8189 driver so I am sure it have nothing to do with the kernel as live boot have the same kernel as the one installed.

Do you install the driver yourself? Otherwise the driver comes with the kernel.

I feel some issue on proxmox network stack once installed. I will install ubuntu server on the device just to make sure it works on my network in and outside DHCP range. then will come back to proxmox inside VM on my laptop as host.

No need to install Ubuntu. just test it from the live installer environment?

OK I AM BLOWN AWAY NOW.
This is Proxmox 8.1.2 on VM on my laptop host. It can ping other pc but not gateway. so It is definitely not pc problem. Something seriously missing on proxmox. Let me try 6.4.1

PVE 6.4 is out of support for a long time, so not a good idea.

There have been issues with RTL network controllers when upgrading to newer Proxmox/kernel versions, but I could not find a report about 8189.
Did you try a different IP address than 192.168.100.2 yet?

 

[spikerguy]

Also sound like traffic between opnSense and Proxmox is not working. What kind of switch is used? Do you maybe have a duplicate MAC or IP in the network?

Yes this is true about traffic, DLINK Unmanaged 16port switch. No duplicate MAC or IP as it works on Live Boot. and I continued with the same IP after reboot too.

The only way to find out is from routers logs.

 

[spikerguy]

Do you install the driver yourself? Otherwise the driver comes with the kernel.

No just using default kernel driver.

No need to install Ubuntu. just test it from the live installer environment?

yes it was just a way to test if that works then it is clear that its not a routing issue.

PVE 6.4 is out of support for a long time, so not a good idea.

Not for production, just to test.

Did you try a different IP address than 192.168.100.2 yet?

Yes, I re-installed and on live boot it got IP from DHCP and I kept the same IP and reboots and lost ping after that


ok found something sensible.
Proxmox bridge refuses to communication with gateway cause it have 00:00:00:00:00 as mac address. Is that a reason ? I hope thats the reason as I am anyways migrating my opnsense so hope it resolves the issue.

Thanks.


UPDATE: Installed Ubuntu on exact same VM and PC and it worked out of the box on static IP. so IDK what is unique in Proxmox that is not allowing to ping the gateway.

Any help is appreciated.

Some Positive update:
When I switch from vmbr to the physical interface then everything works fine on the proxmox host.

root@pve:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 7c:83:34:b0:ee:91 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.167/24 scope global enp1s0
valid_lft forever preferred_lft forever
inet6 fe80::7e83:34ff:feb0:ee91/64 scope link
valid_lft forever preferred_lft forever
3: wlp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether a8:93:4a:df:18:3f brd ff:ff:ff:ff:ff:ff
root@pve:~# ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=0.231 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=0.182 ms
64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=0.168 ms
64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=0.187 ms
^C
--- 192.168.100.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3074ms
rtt min/avg/max/mdev = 0.168/0.192/0.231/0.023 ms
root@pve:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet static
#auto vmbr0
#iface vmbr0 inet static
address 192.168.100.167/24
gateway 192.168.100.1
bridge-ports enp1s0
bridge-stp off
bridge-fd 0

iface wlp2s0 inet manual

 

[spikerguy]

Found the issue.:

When Gateway does not a clean MAC then proxmox does not allow traffic. I added a mac to the interface and that resolved the issue and I migrated my opnsense to a brand new device which have been pending since months. so got it resolved.

Thanks for the responses.