VM can't ping out / no traffic on the fw* interfaces on the host

rdtsupport · Apr 14, 2025

Version: 8.3.5

Problem:

The vm inside sees the configured MAC address off it's interface , so infos are passed to the vm. But the vm can't send data outside.
Even with a correct configured ip / netmask and route inside the vm, tcpdump can't see any traffic what so ever on the fw* interfaces below:

btw: sorry for using <CLASSB> as a visual protection.

sysctl.conf

net.ipv4.ip_forward=1

when the vm is running, the interfaces are created:

Code:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp41s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether a8:a1:59:c0:d0:c0 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether a8:a1:59:c0:d0:c0 brd ff:ff:ff:ff:ff:ff
4: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr104i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ce:b0:82:ad:33:b6 brd ff:ff:ff:ff:ff:ff
5: fwbr104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether aa:2b:30:36:e2:b6 brd ff:ff:ff:ff:ff:ff
6: fwpr104p0@fwln104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether de:ef:67:c0:c1:17 brd ff:ff:ff:ff:ff:ff
7: fwln104i0@fwpr104p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr104i0 state UP mode DEFAULT group default qlen 1000
    link/ether aa:2b:30:36:e2:b6 brd ff:ff:ff:ff:ff:ff

They vanish, if the vm is stopped.

interface config:

Code:

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp41s0
iface enp41s0 inet manual

iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
    address <CLASSB>.54.34/26
    gateway <CLASSB>.54.1
    bridge-ports enp41s0
    bridge-stp off
    bridge-fd 0
    up route add -net <CLASSB>.54.0 netmask 255.255.255.192 gw <CLASSB>.54.1 dev vmbr0

iface vmbr0 inet6 static
    address IPV6ADDR:2/64
    gateway fe80::1

The up route rule was taken to vmbr0 , because the enp41s0 interface was configured by the setup script as the primary interface without a bridge.
I had to create a vmbr0 via UI and change the interfaces.new file to move the rule to the vmbr0 block.

default via <CLASSB>.54.1 dev vmbr0 proto kernel onlink
<CLASSB>.54.0/26 via <CLASSB>.54.1 dev vmbr0
<CLASSB>.54.0/26 dev vmbr0 proto kernel scope link src <CLASSB>.54.34

I have another PVE, but that did not need have this rule, I assume its obsolete anyway:

<CLASSB>.54.0/26 via <CLASSB>.54.1 dev vmbr0

even if it would do something, it can't stop traffic from the vm to disappear on the created fw* interfaces.

I knew, i had such a problem with a bridge before (not pve), but i can't remember how to fix this.

rdtsupport · Apr 14, 2025

Update:

If the vm sends a ping to the gateway x.x.54.1 it gets "Destination Host unreachable"

If the vm uses tcpdump on that interface, it can sniff the entire network traffic from the hosts interface enp41s0.

The host can't see any traffic from the vm on enp41s0 .

How is it possible, that the tcp/ip stack can read the network, but is unable to put a packet on it?????

rdtsupport · Apr 14, 2025

Update:

giving the gateway ip the correct MAC addr ,enables the ping packet to be visible on the hosts interface, but produces no answere.

rdtsupport · Apr 14, 2025

Update:

Ping from VM to Host ip gives:

11:08:56.582295 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo request, id 10, seq 4, length 64
11:08:56.582309 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo reply, id 10, seq 4, length 64
11:08:56.582411 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo reply, id 10, seq 4, length 64

so, we have 2 replies for 1 request. ??

Means also, the vm can put a packet on the interface and the host can see and react to it.

rdtsupport · Apr 14, 2025

it's like the host does not route traffic to the interface for the vms.. even if i add arp entries manually and the host ping uses the tap104i0 interface for the vm directly, it can't receive the arp reply. The vm does receive the icmp request and answeres with a reply.

There is no firewall in the way.

shanreich · Apr 14, 2025

rdtsupport said:
it's like the host does not route traffic to the interface for the vms..

a bridge only operates on layer 2, so it does not route. There's also no need to enable ip_forwarding then.

rdtsupport said:
<CLASSB>.54.0/26 via <CLASSB>.54.1 dev vmbr0

traffic inside the subnet does not need a route, you are telling the PVE host to send all traffic inside that subnet to the gateway - which is most likely not what you want.

rdtsupport said:
But the vm can't send data outside.

Do you mean to the internet, or external networks? did you configure a gateway inside the VM?

rdtsupport · Apr 14, 2025

Just to get sure:

# Generated by iptables-save v1.8.9 on Mon Apr 14 13:36:24 2025
*raw

REROUTING ACCEPT [1737933:614389096]
:OUTPUT ACCEPT [1571662:581378792]
COMMIT
# Completed on Mon Apr 14 13:36:24 2025
# Generated by iptables-save v1.8.9 on Mon Apr 14 13:36:24 2025
*filter
:INPUT ACCEPT [1701909:604146213]
:FORWARD ACCEPT [27471:1273305]
:OUTPUT ACCEPT [1571662:581378792]
COMMIT
# Completed on Mon Apr 14 13:36:24 2025

no firewall active.
vm states firewall=No.
all interfaces allow forwarding

rdtsupport · Apr 14, 2025

shanreich said:
a bridge only operates on layer 2, so it does not route. There's also no need to enable ip_forwarding then.

Compared

working pve: vmbr0 has forwarding enables, other other interfaces have disabled it.
not working pve: i tried to set it the same but, something enabled ip_forwarding on any interface, no matter what i do.

shanreich said:
traffic inside the subnet does not need a route, you are telling the PVE host to send all traffic inside that subnet to the gateway - which is most likely not what you want.

That specific routing has been removed while debugging, no change.

shanreich said:
Do you mean to the internet, or external networks? did you configure a gateway inside the VM?

Status atm:

the host can send and receive anything.

the vm traffic does not show up in vmbr0 tcpdumps
the vm traffic does show up in tap104i0

the vm can receive traffic via tap104i0 when a host route on the host tell it to route traffic to tap104i0.
the vm reacts to traffic it receives "icmp request" with "icmp reply", but the host which sent the request, does not get this. tcpdump can see it on the tap104i0 interface, but that's it.

Interfaces:

Code:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp41s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:a1:59:c0:d0:c0 brd ff:ff:ff:ff:ff:ff
12: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr104i0 state UNKNOWN group default qlen 1000
    link/ether e2:79:06:5b:98:67 brd ff:ff:ff:ff:ff:ff
13: fwbr104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 66:58:79:8e:c7:91 brd ff:ff:ff:ff:ff:ff
14: fwpr104p0@fwln104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9e:7e:45:95:e1:59 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9c7e:45ff:fe95:e159/64 scope link
       valid_lft forever preferred_lft forever
15: fwln104i0@fwpr104p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr104i0 state UP group default qlen 1000
    link/ether 66:58:79:8e:c7:91 brd ff:ff:ff:ff:ff:ff
16: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:a1:59:c0:d0:c0 brd ff:ff:ff:ff:ff:ff
    inet <CLASSB>.54.34/26 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 2a01:4f9:5a:xxxx::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::aaa1:59ff:fec0:d0c0/64 scope link
       valid_lft forever preferred_lft forever

route:

default via <CLASSB>.54.1 dev vmbr0 proto kernel onlink
<CLASSB>.54.0/26 dev vmbr0 proto kernel scope link src <CLASSB>.54.34
<CLASSB>.54.29 dev tap104i0 scope link

tcpdump:

root@standby /etc/sysctl.d # tcpdump -n -i tap104i0 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on tap104i0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:30:46.699270 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo request, id 53386, seq 61, length 64
13:30:46.699339 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo reply, id 53386, seq 61, length 64
13:30:47.723188 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo request, id 53386, seq 62, length 64
13:30:47.723276 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo reply, id 53386, seq 62, length 64
13:30:48.747160 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo request, id 53386, seq 63, length 64
13:30:48.747325 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo reply, id 53386, seq 63, length 64
13:30:49.771260 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo request, id 53386, seq 64, length 64
13:30:49.771345 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo reply, id 53386, seq 64, length 64
13:30:50.795262 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo request, id 53386, seq 65, length 64
13:30:50.795323 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo reply, id 53386, seq 65, length 64
13:30:51.819247 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo request, id 53386, seq 66, length 64
13:30:51.819380 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo reply, id 53386, seq 66, length 64
13:30:52.843213 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo request, id 53386, seq 67, length 64
13:30:52.843382 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo reply, id 53386, seq 67, length 64
13:30:53.867253 IP <CLASSB>.54.34 > <CLASSB>.54.29: ICMP echo request, id 53386, seq 68, length 64
13:30:53.867365 IP <CLASSB>.54.29 > <CLASSB>.54.34: ICMP echo reply, id 53386, seq 68, length 64
(but ping itself does not get the answere, even if using this tap specific interface)

@Shanrich:

which info you to shine some light on it? Screenshots? files ..

shanreich · Apr 14, 2025

rdtsupport said:
That specific routing has been removed while debugging, no change.

you should not need to add any custom routes or arp entries, the following (automatically created!) route is sufficient:

<CLASSB>.54.0/26 dev vmbr0 proto kernel scope link src <CLASSB>.54.34

Try removing the custom route from your /etc/network/interfaces (post-up) - then could you maybe reboot the host, so any possible lingering changes are removed?

After rebooting, post the output of the following commands:

Code:

ip a
ip r
cat /etc/network/interfaces
iptables -t nat -L

Also, please post the network configuration from inside the VM

rdtsupport · Apr 14, 2025

Rebooting .. this is set manually inside the vm:

Host:

ip a:

Code:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp41s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:a1:59:c0:d0:c0 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:a1:59:c0:d0:c0 brd ff:ff:ff:ff:ff:ff
    inet <CLASSB>.54.34/26 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 2a01:4f9:5a:xxxx::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::aaa1:59ff:fec0:d0c0/64 scope link
       valid_lft forever preferred_lft forever
4: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr104i0 state UNKNOWN group default qlen 1000
    link/ether 02:bc:7b:3e:36:d5 brd ff:ff:ff:ff:ff:ff
5: fwbr104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 82:d2:95:0f:e9:3c brd ff:ff:ff:ff:ff:ff
6: fwpr104p0@fwln104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 16:10:9e:b8:2c:50 brd ff:ff:ff:ff:ff:ff
7: fwln104i0@fwpr104p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr104i0 state UP group default qlen 1000
    link/ether 82:d2:95:0f:e9:3c brd ff:ff:ff:ff:ff:ff

ip r:

Code:

default via <CLASSB>.54.1 dev vmbr0 proto kernel onlink
<CLASSB>.54.0/26 dev vmbr0 proto kernel scope link src <CLASSB>.54.34

/e/n/interfaces:

Code:

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp41s0
iface enp41s0 inet manual

iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
    address CLASSB.54.34/26
    gateway  CLASSB.54.1
    bridge-ports enp41s0
    bridge-stp off
    bridge-fd 0
#    up route add -net  CLASSB.54.0 netmask 255.255.255.192 gw CLASSB.54.1 dev vmbr0

iface vmbr0 inet6 static
    address 2a01:4f9:5a:xxxx::2/64
    gateway fe80::1

Iptables

Code:

 iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

In this state, after rebooting, not only restarting the network as last time, and fixing corosync.conf again we have the following situation:

Inside the VM
- i see an arp addr for the host
- i can ping the host
- i see an incomplete arp addr for the gateway
- i can't ping the gateway or anything outside of the hist

(which is, alltogether, much better than before)

Outside the vm on the host:

- i see an arp addr for the vm in vmbr0
- i can ping the vm and get an answere
- tcpdump sees all traffic on tap104i0 , as it's on vmbr0 and enp41s0
- i see arp request from the vm on tap104i0, but no answere
- i see the arp request from the vm on vmbr0 AND the reply to it.

Conclusion:

ATM arp is not forwared INTO the vm tap interface

rdtsupport · Apr 14, 2025

Addon:
root@standby ~ # cat /proc/sys/net/ipv6/conf/*/forwarding
1
1
1
1
1
1
1
1
1
root@standby ~ # cat /proc/sys/net/ipv4/conf/*/forwarding
1
1
1
1
1
1
1
1
1
root@standby ~ # cat /proc/sys/net/ipv4/ip_forward
1

coming from /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

rdtsupport · Apr 14, 2025

If anyone ever has the same issues.. do not use the Hetzner proxmox installscript .. you can see what you have to deal with later

shanreich · Apr 14, 2025

rdtsupport said:
If anyone ever has the same issues.. do not use the Hetzner proxmox installscript .. you can see what you have to deal with later

Is this a public IP on a hetzner host?

rdtsupport · Apr 14, 2025

I ordered a server with the .54.34 as host ip.
I also ordered an additional ip 54.29 for the server to be used on the vm

pls focus on this:

- i see arp request from the vm on tap104i0, but no answere
- i see the arp request from the vm on vmbr0 AND the reply to it.

the ip seems to be valid inside the hetzner system or it would not gotten a arp reply for the gateway arp request

rdtsupport · Apr 14, 2025

Now it gets funny.. a ping from an EXTERNAL server is going inside the vm, gets replied in the tap103i0, is visible in vmbr0 and on enp41s0.

but never reaches the pinging server

rdtsupport · Apr 14, 2025

I wrote to hetzner to add the mac to the filter list. Maybe that helps.

rdtsupport · Apr 14, 2025

ARGS.. you need to request a seperate mac for vm ips.

*SOLVED*

rdtsupport · Apr 14, 2025

THX all the affort.

Search

Search

VM can't ping out / no traffic on the fw* interfaces on the host

rdtsupport

New Member

rdtsupport

New Member

rdtsupport

New Member

rdtsupport

New Member

rdtsupport

New Member

shanreich

Proxmox Staff Member

rdtsupport

New Member

rdtsupport

New Member

shanreich

Proxmox Staff Member

rdtsupport

New Member

rdtsupport

New Member

rdtsupport

New Member

shanreich

Proxmox Staff Member

rdtsupport

New Member

rdtsupport

New Member

rdtsupport

New Member

rdtsupport

New Member

rdtsupport

New Member

We value your privacy