ESXi to Proxmox - Storage is not online (500)

[cust0m]

Hi,

Is there a way to troubleshoot the following further (e.g. are there any log files which would help me to solve that) ?

When I want to add an ESXi storage via "Datacenter -> Storage -> Add -> ESXi", I always get the following error:
create storage failed: storage 'esxi' is not online (500)

I have of course checked the credentials, the ESXi server can be reached via the IP (same subnet, no firewall restrictions) and I have also activated the "Skip Certificate Verification" option.

Best regards,
cust0m

 

[mmccarn]

I saw this after a failed import from ESXi 6.7.0U3 using pve-esxi-import-tools v0.6.0. Imports failed around 14GB, and seemed to trigger a firewall setting on the ESXi side that blocked access from PVE with a 500 error.

v0.6.1 and v0.7.0 of the tools solve the import issue with ESXi 6.7 -- but you need to delete the esxi connections, reboot the PVE host (or restart some services) then re-connect the esxi host.

 

[cust0m]

I found something out:

For my test scenario, esxi is running on proxmox (nested virtualozation).

And while the esxi server is pingable from outside (client in the same subnet as the proxmox node), it is not pingable from the proxmox node itself.

That is not the case for other VMs/containers on this node.

The firewall checkbox on the virtual nic of the proxmox server is not ticked...

Any ideas?

 

[xpaz]

same issue, following. VMware ESXi seems to have a problem with the networking in the nested virtualisation under proxmox.

 

[gfngfn256]

That is not the case for other VMs/containers on this node.

Does that include; you can ping from the VM to the Esxi (virtualized VM?) same node?

 

[cust0m]

It is also not possible to ping from a VM (virtualized under Proxmox) to the ESXi host. The VM is running on the same Proxmox node that can't ping the ESXi host.

The following works (everything in the same subnet):

physical host → ESXi Host
physical host → ESXi VM
VM (Proxmox) → VM (ESXi)
VM (ESXi) → VM (Proxmox)

But it's not possible to ping the ESXi host from the Proxmox node (that's why I can't mount the storage, I guess) or to ping from a VM running on the same Proxmox node to the ESXi host.

 

[gfngfn256]

physical host → ESXi Host
physical host → ESXi VM
VM (Proxmox) → VM (ESXi)
VM (ESXi) → VM (Proxmox)

Something really odd in your NW obviously.

Assuming pinging can be done both ways - I can see a route to take to get from VM (Proxmox) → ESXi Host although you say it's not possible:

VM (Proxmox) → VM (ESXi) → physical host → ESXi Host

(maybe could be even shorter - if VM (ESXi) → ESXi Host is pingable but you don't provide that info).

The only plausible reason for this behavior, will have to come down to either some odd firewall/iptables behavior blocking ICMP selectively somewhere in a router/switch/pc/server along the route, or some point in the middle shares more than one network.

Assuming you've got SSH enabled in all the above - you should be able to use SSH forwarding to connect through SSH the above suggestion.

You may need to do a traceroute on the above connection(s) you mention that work, to actually see how they are connecting.

 

[cust0m]

As I wrote, all hosts are in the same subnet and there is no firewalls/router in between.
The ESXi host is running on the Proxmox node itself (nested virtualization).
Traceroute isn't showing anything, since there are no further hosts in between.

I solved it for myself, I just migrated the VM with ESXi to another Proxmox host in the same subnet.
Then everything is working as expected, but not when using nested virtualization.

And as @xpaz wrote, I am not the only one with this behavior.

 

[gfngfn256]

I solved it for myself, I just migrated the VM with ESXi to another Proxmox host in the same subnet.
Then everything is working as expected, but not when using nested virtualization.

Glad you got it working - but it still doesn't solve the question of the above mentioned oddity.
Also you're still using nested virtualization on the ESXi inside Proxmox.
I wonder if on the other Proxmox host you moved the ESXi to, can you ping from that host to the ESXi running on it.

 

[cust0m]

Yes, nested virtualization is still enabled on the VM that runs esxi on the other Proxmox node.

From this other Proxmox node, the ESXI host (that runs on it) is pingable.

The quesion is, what could cause this strange behavior on the first Proxmox node.

The test systems are running different versions (Proxmox 7 on the working one, 8 on the non-functioning one).

 

[gfngfn256]

(Proxmox 7 on the working one, 8 on the non-functioning one)

I assume you've compared the firewall rules/settings on both hosts & have not found them to be different.

If this is indeed the case - then this could prove to be linked to the PVE overhaul concerning Firewall & SDN, see here & also in the changelogs further along.

BTW what is the output from both nodes of:

pve-firewall status

 

[cust0m]

It's just the default firewall settings on both hosts, no rules defined under <node> → Firewall.
Settings within Firewall → Options are identical.
pve-firewall status shows Status: disabled/running on both hosts.

 

[gfngfn256]

It's just the default firewall settings on both hosts, no rules defined under <node> → Firewall.
Settings within Firewall → Options are identical.
pve-firewall status shows Status: disabled/running on both hosts.

Then assuming the VMs (ESXi) are identical on both hosts - something else within PVE is causing this difference. This needs investigating - especially if it turns out something within the PVE 7 to 8 version change.

 

[ChadAndersen]

I had the same issue with my ESXi running in my Proxmox server - I changed the network adapter from "VMware vmxnet3" to "Intel E1000". That resolved the host not being able to ping/mount volumes from the ESXi guest. I still have some issues on importing VMs from the ESXi system to Proxmox.

 

[piov_voip]

I had the same issue with my ESXi running in my Proxmox server - I changed the network adapter from "VMware vmxnet3" to "Intel E1000". That resolved the host not being able to ping/mount volumes from the ESXi guest. I still have some issues on importing VMs from the ESXi system to Proxmox.

Be wary of rolling this one out IF you have a single NIC. I haven't fully investigated my issue. My VM import hit the OP's problem, I attempted to import the vm and changed adapter from vmware vmxnet3 to intel e1000 & that's it. I cannot hit the management ip.

I'm brand new to Proxmox, so still learning the ropes; esxi is a piece of cake compared to this

 

[ChadAndersen]

Be wary of rolling this one out IF you have a single NIC. I haven't fully investigated my issue. My VM import hit the OP's problem, I attempted to import the vm and changed adapter from vmware vmxnet3 to intel e1000 & that's it. I cannot hit the management ip.

I'm brand new to Proxmox, so still learning the ropes; esxi is a piece of cake compared to this

Perhaps you missed that I changed the Virtualized ESXi host's nic type in Proxmox from "VMware vmxnet3" to "Intel E1000" - not the Proxmox nic. If it didn't work, I could always change it back. My virtual ESXi host is just to grab non-running vmware VMs off the iSCSI storage.

I found the Proxmox ESXi storage mount for importing vmware VMs didn't work reliable enough to use. I ended up using a process like this post https://forum.proxmox.com/threads/migration-from-vmware-esxi-to-proxmox.130079/post-570699

 

[piov_voip]

Perhaps you missed that I changed the Virtualized ESXi host's nic type in Proxmox from "VMware vmxnet3" to "Intel E1000" - not the Proxmox nic. If it didn't work, I could always change it back. My virtual ESXi host is just to grab non-running vmware VMs off the iSCSI storage.

I found the Proxmox ESXi storage mount for importing vmware VMs didn't work reliable enough to use. I ended up using a process like this post https://forum.proxmox.com/threads/migration-from-vmware-esxi-to-proxmox.130079/post-570699

Update: As I mentioned, I'm new to Proxmox. I'm still not fully sure what occured, but I set the CPU type to 'host' and didn't alter the NIC... reached 100%! on my test (one of my prod) vms

 

[cody0922]

had the same problem (nested esxi vm on proxmox host, couldn't ping esxi vm from the host or connect to datastore to import vmware vm). the esxi vm network was on the same vmbr0 as the proxmox mgmt interface.

the fix i found was adding another nic and creating another vmbr for that nic, and setting the esxi vm to use that vmbr. in my case, i just used a usb nic connected to lan vlan port on a switch. no need to configure an IP on the new vmbr.