Gaps in Proxmox GUI graph

[wahmed]

Anybody experienced frequent gaps in the GUI graph as in the following screenshot?


It started after last batch of updates of all Proxmox nodes. Gaps is consistent across all nodes. They are not in same time period. Seems to occur in random times but on all nodes. Any idea how i can pin point the cause?

It also seems to interacting with storage somehow. For example, if i am trying to create a new VM, on Hard Disk portion the storage drop down menu seems to be disabled for sometime then gets enabled on its own. Syslog says nothing.

 

[dietmar]

Maybe one storage is not online? Test with

# pvesm status

 

[wahmed]

Maybe one storage is not online? Test with

# pvesm status

pvesm shows all Storage is online. Gap still continues in random interval.

 

[davlaw]

Not sure what to add, but your graph shows one node. Do all nodes show same gap@time, or is it just affecting one node?

 

[wahmed]

Not sure what to add, but your graph shows one node. Do all nodes show same gap@time, or is it just affecting one node?

Yes all nodes shows same gap. I have 11 nodes in the cluster. Only showed one node as sample.

Looking back at our cluster maintenance log, it also seems that the issue started right around when we also setup 2 of the nodes as Gluster with ZFS as underlying platform. Its a 2 replica gluster setup sharing the same 20gbps Infiniband network with ceph public network. I have some live VMs in operation in this gluster so cannot just take it offline to see if that is the one causing issue. But i will if it is necessary. Any idea if that could be causing the issue? I do not see how it could related specially when that 20gbps network bandwidth is not even being consumed more than 2 gbps.

 

[mir]

Could it be a timeout issue with the Gluster storage? Gluster on top of ZFS in production sounds scary to me;-)

 

[wahmed]

Could it be a timeout issue with the Gluster storage? Gluster on top of ZFS in production sounds scary to me;-)

I know what you mean!
I tested Proxmox+Gluster+ZFS for several weeks before bringing it to production. I dont recall seeing this issue in that test cluster. But that cluster also did not have any real life load on it.

We have dozens of VM running in this cluster including 2 VDIs where about 3 dozens users regularly logs in to virtual desktops. But havent heard from anybody losing connection momentarily. If we lost connection to storage randomly then these people would have been interrupted.
Is it possible that pvestatd being conflicted with something which causing these gaps?

 

[mir]

By timeout I meant that pvestatd might once in a while timeout when requesting updates from Gluster storage which could cause the gaps.