Hello,
I would like to report and discuss a behavior I have observed in a Proxmox VE 9.2 cluster and ask whether this could be related to the way pvestatd refreshes storage status/statistics.
Environment
I have a 5-node Proxmox VE cluster.All five nodes have identical hardware and are fully updated. The current software versions are:
- Proxmox VE: 9.2.0
- PVE Manager: 9.2.9
- Kernel: 7.0.14-9-pve
- pve-esxi-import-tools: 1.0.1
- Corosync: 3.1.10-pve3
There are five ESXi-type storages configured in Proxmox:
- Four storages point directly to individual ESXi hosts located in the same subnet as the Proxmox cluster.
- The fifth storage points to a vCenter Server located in a different subnet and in a geographically different data center.
- 6 ESXi hosts in the vCenter's local subnet/data center.
- 4 additional ESXi hosts in the same subnet as the Proxmox cluster.
Background: why I configured the ESXi storages
I initially had a single ESXi storage configured against the vCenter.However, when accessing this storage from Proxmox, I noticed significant delays.
For example, when opening the import workflow and asking Proxmox to list the available VMs, the request could take a considerable amount of time and eventually return a connection error.
If I waited and repeated the operation, the VM list could eventually appear.
However, when selecting a VM from the list and proceeding with the import, Proxmox needed to query the VM's information again in order to display its hardware configuration. At that point I would again get a connection error.
In practice, I was unable to reliably import a VM through the vCenter-based ESXi storage.
This behavior was particularly noticeable from the Proxmox cluster because the vCenter is located in another data center/subnet and there is additional network latency between the two sites.
Because of this, I configured individual ESXi storages pointing directly to the ESXi hosts.
Those direct ESXi connections are much more responsive because the ESXi hosts are in the same subnet as the Proxmox cluster.
The behavior I observed
For several days I had been noticing an unusual behavior during maintenance windows involving Proxmox node reboots.When a node was rebooted, another node in the cluster — specifically the node I use as a test node and which had the vCenter-based ESXi storage configured — would sometimes appear in the Proxmox GUI with a grey/unknown status.
The node itself was actually running normally.
The interesting part was that the affected node would remain grey/unknown for approximately 15 minutes, and would eventually return to green/online without any intervention.
I initially suspected a problem with the node itself, pvestatd, corosync, or the cluster communication.
However, today's maintenance window allowed me to reproduce the behavior in a much more controlled way.
Test performed today
The cluster consists of five nodes:- node1
- node2
- node3
- node4
- node5
I then performed the following test.
1. Enable the ESXi storages
I enabled:- On node4: an ESXi storage pointing directly to an ESXi host in the same subnet as the Proxmox cluster.
- On node5: the ESXi storage pointing to the remote vCenter.
node4 → direct ESXi host → same subnet / low latency
node5 → vCenter → different subnet/data center / higher latency
2. Reboot node4
I then rebooted node4.Node4 came back normally.
However, while node4 was rebooting / coming back online, node5 became grey/unknown in the Proxmox GUI.
Node5 itself had not been rebooted.
This is the part that caught my attention.
After approximately 15 minutes, node5 returned to green/online by itself.
Why I suspect the vCenter-based storage
The important observation is that node5 was the only node in this test with the ESXi storage pointing to the remote vCenter.The other ESXi storages point directly to ESXi hosts in the same subnet as the Proxmox cluster.
Therefore, my current hypothesis is that the problem may not be related to the node itself, but rather to the time required to obtain information from the vCenter-based storage.
My understanding is that pvestatd periodically collects status/statistics information for nodes and storages.
If a storage operation involves communicating with a remote vCenter, and that operation is slow or temporarily blocked because of network latency, I wonder whether this can delay or block the status/statistics update sufficiently to make the node appear as grey/unknown in the GUI.
In other words, I am wondering whether something similar to this could be happening:
Proxmox node
|
| pvestatd / storage status query
|
v
ESXi storage
|
v
remote vCenter
|
| network latency / delayed response
|
v
vCenter / ESXi inventory
While this query is taking place, the node's status information may not be refreshed within the expected interval, causing the GUI to display the node as unknown.
Additional observation
I also tested the behavior with the vCenter-based storage enabled on node5 while rebooting node4.Node4, which had the direct ESXi storage configured, did not remain grey after reboot. It came back normally after the expected boot time.
Node5, which had the remote vCenter storage configured, was the node that became grey.
This makes me think the difference between the two storage configurations may be relevant.
The hardware is identical across all nodes, and all nodes are running the same Proxmox/kernel versions, so I do not currently have a reason to suspect a hardware or software-version difference between the nodes.
Previous observations with the vCenter storage
The storage pointing to the remote vCenter has also shown significantly different behavior when used for VM import.From the Proxmox GUI:
- I request the list of VMs from the vCenter-based ESXi storage.
- The request takes a long time.
- Eventually Proxmox reports a connection error.
- If I wait and try again, the VM list may eventually appear.
- I select a VM and start the import process.
- Proxmox then needs to query the VM again to obtain its hardware information.
- At this point, the GUI again reports a connection error and never reaches the point where the VM hardware information is displayed.
These direct connections work much better for my use case.
Current situation
As I now have direct access to the ESXi hosts that I need for the migration process, I no longer need the vCenter-based ESXi storage for VM imports.I have therefore removed the vCenter-based ESXi storage from the Proxmox clusters.
I am nevertheless interested in understanding the behavior because it seems possible that the storage configuration was indirectly affecting the status reporting of the node.
Questions
I would appreciate feedback from the Proxmox developers/community on the following:- Can a slow or unreachable ESXi/vCenter storage operation affect pvestatd enough to cause a node to appear grey/unknown in the GUI?
- Does pvestatd perform storage status/statistics queries in a way that could be affected by a high-latency remote vCenter connection?
- Could a delayed storage query cause the node's status information to become stale and therefore make the GUI display the node as unknown?
- Is there a timeout mechanism for ESXi/vCenter storage queries that prevents a slow remote API call from affecting the normal node status reporting?
- Is there a recommended way to diagnose this relationship, for example by tracing pvestatd activity or identifying which storage operation is blocking/delaying the status update?
I am not claiming that the vCenter storage is definitely the cause. However, after today's controlled test, the correlation seems strong enough that I would like to understand what is happening internally.
Thanks in advance for any feedback.
