No access to nested ESXi 7.0.3 host after upgrade to 8.1 - Connection between nested host works!

[steveheart]

Hello everyone,

after updating to Proxmox VE version 8.1, I can no longer access my nested ESXi hosts. With Proxmox VE version 8.0 everything still worked perfectly. The strange thing is that I can communicate between the ESXi hosts: ssh from esxi1 to esxi2 works!

The ESXi hosts have vSphere 7.0.3 installed.

Thanks for any hint.

Steven

 

[troythector]

I'm having the same problem with my nested ESXi 8.0.0 (VMKernel Release Build 20513097) environment after upgrading to Proxmox VE 8.1.



The above setup worked before the PVE update. I have tried changing the interface Model to "Intel E1000" and "VirtIO (paravirtualized)" but these adapters are not supported.

 

[steveheart]

Hello everyone,
now comes the interesting part:

I can access the running VMs on the nested host but I can't access the ESXi management console. No firewall active. Something seems to be blocking the traffic to the hypervisor itself.

 

[rellek74]

There's a few threads with this issue. VLAN Networking is screwy after updates. I think it has something to do with drivers.

Are these VM's or LXC? My issues were mainly with LXC, I was able to get the VM's talking. I had to disable VLAN aware, reboot, RE-enable it, Reboot, and then was able to get some working.

 

[steveheart]

I do have ESXi running nested in two VMs. I can access the VMs running on ESXi but I can't access the ESXi management console, listening on port 443 nor can I access port 902 for NBD access.

 

[rellek74]

I do have ESXi running nested in two VMs. I can access the VMs running on ESXi but I can't access the ESXi management console, listening on port 443 nor can I access port 902 for NBD access.

Do you have VLAN's? Make sure your linux bridge is set to VLAN aware and give it a reboot. Helped with a few of mine, but not all.

 

[rellek74]

This may help too:

ifdown --force -vvv <iface>
ip address flush dev <iface>
ip link set <iface> down
ifup -vvv <iface>

 

[steveheart]

Thanks, I do have no VLANs configured. The VMs network devices are attached to a bridge therefore the interface are name tapXXXi0 and then your commands will not work.
The interfaces are up, because otherwise I would not have access to the running VMs on the nested ESXi host. Strange....

 

[xybor]

Thanks, I do have no VLANs configured. The VMs network devices are attached to a bridge therefore the interface are name tapXXXi0 and then your commands will not work.
The interfaces are up, because otherwise I would not have access to the running VMs on the nested ESXi host. Strange....

Hi,
Have you figured it out?
I'm facing the same problem...

 

[steveheart]

Hi @xybor,
no still having the issue, even with the latest Proxmox update applied. The problem is somehow on vmkernel, it's not getting the MAC address from the requestor.

What I can do
- Access the VMs running on the ESXi host from the "outside world" using ssh or RDP or https
- Access the ESXi server from another ESXi server attached to the same bridge works. SSH or ping

When I ping the ESXi host for example from a windows system, the MAC address get's registered on the Windows host, on the ESXi host you see after entering the command "esxcli network ip neighbor list" that the requesting IP address is visible, but no MAC address is given and the Type column shows "invalid".

So somehow something is blocking here. But, beside of the Proxmox updates, there were no changes in this environment, which runs since more than a year (started with version 7).

 

[steveheart]

According to the roadmap this has changed for version 8.1

Proxmox Server Solution GmbH, the company behind Proxmox VE development and infrastructure, was assigned an official Organizationally Unique Identifier (OUI) BC:24:11 from the IEEE to use as default MAC prefix for virtual guests.


Looking at the following tcpdump command I see (oui Unknown) messages

tcpdump -i <tap interface of esxi host> -vv arp

Log entries for ping from windows-vm from esxi host:

Request who-has <ip-address.windows-vm> tell <ip-esxi-host>, length 46
Reply <ip-address.windows-vm> is-at bc:24:11:**:**:** (oui Unknown), length 28


Ping from esx host to windows-vm shows nothing in tcp dump.

On the Proxmox host and on the Windows VM I see the vmkernel port MAC address from the esxi host! (d6-f8-5d-**-**-**).

Even when changing the ESXi vmxnet3 adapter's MAC address to bc:24:11::**:**:** and also set the option

esxcfg-advcfg -s 1 /Net/FollowHardwareMac

to use the same MAC as the HW, doesn't change anything in the behavior.

 

[tchuyev]

Exact same problem here with ESXi 7.0.3.
Any news on this issue please? Thanks.

 

[steveheart]

It must me something around the vmxnet3 driver. I quickly spinned up an ESXi 6.7 host with an E1000 NIC for the vmkernel port. I could access everything.

 

[akidarsa]

Same issue with ESXi 8.0 on Proxmox 8.1.3 with Kernel 6.5.11-7-pve.

Is this issue in the bug report or anything we can do so that it can be taken a look at by someone that might be able to fix it?

 

[itobin]

anyone got this working yet?

 

[saltybob]

New to the forum, but also really interested in seeing this get resolved. Any way we can help?

 

[B-C]

I set mine up a little different but seeing the same thing...

3 Hosts - SDN Setup as VXLAN & Ceph Storage setup between them.
VMs
1x OpenVPN Host
- WAN - On Bridge Interface with Hosts
- LAN - on VXLAN
1x linux client - can ping both OpenVPN and out the NAT to other network
Cannot ping ESXi Hosts
2x ESXi v7.0.3 - Setup with VMX3 Nics

Able to ping ESXi <> ESXi (hosts on on different nodes in the cluster as well)
Can set ESXi to dhcp - will get a dynamic IP from OPNSense without issue, but cannot ping it either direction from ESXi > Firewall or Firewall > ESXi

ESXis are located on different Hosts and still they can ping each other without any issue. Can migrate them to the same host as the Firewall & Client so all on the same host and no difference.

Being these are fresh ESXi v7 installs there are no vm's and no good way to spinup a vm as they're behind a virtualfirewall...
really only a vpn inbound to that network via the virtual firewall but if that firewall can't hit them the vpn clients wouldn't be able to either.

Like others have clearly identified - the issue seems to be somewhere with ARP issue in vSwitch0 and vmx3net getting filtered somewhere
the funky part is DHCP working on the broadcast side but then getting hung up with actual communication.

 

[klwood2000]

I'm seeing the same issue here. I have Proxmox 8.2 running kernel 6.5.13-5-pve. I have 3x 8.01u1 nested esxi hosts running. I originally set them up with vmxnet3 adapters which I could not ping from a windows VM on the same bridge, but could ping from my laptop. I have tried the e1000e and e1000. e1000e responds to ping, but e1000 is not supported in esxi. I have a vCenter up and running nested inside esxi and it works fine. I have a windows VM running Veeam running in proxmox. It will attach to the vCenter, but will not recognize the hosts. I'm not running any vlans and I have turned on vlan aware for my bridge and no changes. Any updates to help with this would be great.

 

[klwood2000]

As an update, I finally fixed proxmox to boot the latest kernel 6.8.4-2 and that still hasn't fixed the issue.

 

[mike.spragg]

Oh drat. I tried Proxmox8.2.2 [Linux 6.8.4-2-pve (2024-04-10T17:36Z)] on VMWare workstation as a test - set as nested VMWare ESXi v8 or above and *everything* worked perfectly. I then setup a new install of pve on my VMWare ESXi 8.U2 (simply testing but better kit to work with) and, again, all worked fine except as above - couldn't make it see the network - it gets an IP from DHCP (so is seeing other hosts/dhcp server) and the right subnet/gateway are all correct - but the Win2022 VM (which installed fine - all with the virtIO drivers (and the drivers installed inside windows, there are no issues with devices in CP) says "no internet access" and unidentified network. It can't ping it's own gateway. I can't change the VMXNET3 adapater as that's the only one available for this VM when set to ESXi 8> so a bit stopped. Interestingly, I was testing Xenserver 8 and it had exactly the same issue so I'm guessing it's the VMXNET3 that's the issue here. Bit stuck at the mo - but still preferring Proxmox over Xenserver as seems much more highly configurable.