ESXi migrated to Proxmox -> Ping: Nexthop issue

[AnM]

A few weeks ago I started migrating from ESXi to Proxmox. Since then I
have some network issues that I fail to explain. Any advice highly
appreciated


My (reduced) network layout is like

PFSense ←> Zyxel GS1920 ←>
ESXi
Proxmox

There is a ton of other units, but none that was touched during the migration test.
Cables to each server has been tested
Different switch ports were tested

The issue at hand:


ESXi 10.3.0.101

VMs in ESXi on VLAN 3

• can Ping each other
• can Ping ESXi
• can Ping VMs on Proxmox
• cannot Ping Proxmox

ESXi cannot ping Proxmox
ESXi can ping PFsense
VM in ESXi on VLAN 5 CAN ping Proxmox and ESXi


Proxmox 10.3.0.100

VMs in Proxmox on VLAN 3

• can Ping each other
• can Ping Proxmox
• can Ping VMs on Proxmox
• cannot Ping ESXi (Error from ping: From 10.3.0.100: icmp_seq=1 Redirect Host(New nexthop: 10.3.0.101)

Proxmox cannot ping ESXi
Proxmox can ping PFSense


PFSense

can ping ESXi
can ping proxmox, for a while. Then “Redirect Host(New addr: ”
has been running for years


The ESXi server is the oldest and have co-existed with other ESXi servers for many years

The switch has not been updated since the migration to proxmox, so network configuration is the same


My Proxmox network interfaces file

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno1.2
iface eno1.3
iface eno1.5

auto vmbr0v2
iface vmbr0v2 inet manual
bridge-ports eno1.2
bridge-stp off
bridge-fd 0

auto vmbr0v5
iface vmbr0v5 inet manual
bridge-ports eno1.5
bridge-stp off
bridge-fd 0

auto vmbr0v3
iface vmbr0v3 inet static
address 10.3.0.100/16
gateway 10.3.0.1
bridge-ports eno1.3
bridge-stp off
bridge-fd 0

It feel like I have misunderstood something reg. vmbr0v3, but I fail to see it
Perhaps ESXi "forgave" a switch configuration issue ?

 

[shanreich]

Seems like you are getting an ICMP redirect message, can you try and send me a tcpdump of the ping attempts from the Proxmox host?

 

[AnM]

Yes, I get ICMP redirects, but I do not understand what is wrong

Ping from Proxmox -> ESXi
ping 10.3.0.101

PING 10.3.0.101 (10.3.0.101) 56(84) bytes of data.
From 10.3.0.100 icmp_seq=1 Destination Host Unreachable

Ping from PFSense -> ESXi
64 bytes from 10.3.0.101: icmp_seq=0 ttl=64 time=0.513 ms
92 bytes from 10.3.0.100: Redirect Host(New addr: 10.3.0.101)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 0054 8420 0 0000 3f 01 e307 10.3.0.1 10.3.0.101

Tcpdump on Proxmox
tcpdump -v -n -i eno1 icmp

tcpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes

11:56:19.319715 IP (tos 0x0, ttl 64, id 33824, offset 0, flags [none], proto ICMP (1), length 84)
10.3.0.1 > 10.3.0.101: ICMP echo request, id 53607, seq 1, length 64

11:56:19.319785 IP (tos 0xc0, ttl 64, id 38608, offset 0, flags [none], proto ICMP (1), length 112)
10.3.0.100 > 10.3.0.1: ICMP redirect 10.3.0.101 to host 10.3.0.101, length 92
IP (tos 0x0, ttl 63, id 33824, offset 0, flags [none], proto ICMP (1), length 84)
10.3.0.1 > 10.3.0.101: ICMP echo request, id 53607, seq 1, length 64

11:56:20.320688 IP (tos 0x0, ttl 64, id 1359, offset 0, flags [none], proto ICMP (1), length 84)
10.3.0.1 > 10.3.0.101: ICMP echo request, id 53607, seq 2, length 64

 

[shanreich]

How do the routes on the PVE host and the ESXi host look like? It seems like the traffic from ESXi <-> Proxmox is going through the gateway, who is telling you to just directly send it since you're both on the same LAN.

 

[AnM]

Networking can be quite tricky

More ping tests
Ping from VM on ESXi -> VM on Proxmox
PFSense tcpdump icmp shows no data (VMs on same VLAN and same subnet)
Ping is successful

Ping from ESXi -> Proxmox
PFSense tcpdump icmp shows no data (VMs on same VLAN and same subnet)
Proxmox tcpdump icmp shows no data (checked eno1 and eno1.3)
Ping is NOT successful

Ping from VM on ESXi -> proxmox
PFSense tcpdump icmp shows no data
Ping is NOT successful


ESXi route
esxcfg-route:
VMkernel default gateway is 10.3.0.1

esxcli network ip route ipv4 list
Network Netmask Gateway Interface Source
-------- ----------- -------- --------- ------
default 0.0.0.0 10.3.0.1 vmk0 MANUAL
10.3.0.0 255.255.0.0 0.0.0.0 vmk0 MANUAL


Proxmox ip route
default via 10.3.0.1 dev vmbr0v3 proto kernel onlink
10.3.0.0/16 dev vmbr0v3 proto kernel scope link src 10.3.0.100


My gut tells me this is a VLAN tagging issue, but I don't see it. Perhaps blinded
since it "worked before proxmox".

If I interpret you correctly, there is no obvious fault with my interfaces file ?