One Promox showing all question marks on machines

[james woods]

Hi there,

so we have a cluster of 4 proxmox nodes running 5.4-13 and one of the nodes keeps showing question marks for all of the machines. When this happens, the graphs for the machines just show as loading.

It looks like this:


This also happened when we were on version 4. Running sudo service pvestatd restart fixes the issue, but only very briefly, a couple of minutes later it shows up like this again. This keeps happening until the node is fully restarted. I can SSH onto the node as normal and pvesm returns and shows the storages as active.

It hasn't happened to other nodes thus far. Could you offer any advice on how to find out what is causing this?

Thank you

 

[adamb]

Hi there,

so we have a cluster of 4 proxmox nodes running 5.4-13 and one of the nodes keeps showing question marks for all of the machines. When this happens, the graphs for the machines just show as loading.

It looks like this:
View attachment 12202

This also happened when we were on version 4. Running sudo service pvestatd restart fixes the issue, but only very briefly, a couple of minutes later it shows up like this again. This keeps happening until the node is fully restarted. I can SSH onto the node as normal and pvesm returns and shows the storages as active.

It hasn't happened to other nodes thus far. Could you offer any advice on how to find out what is causing this?

Thank you

I typically hit this due to a storage issue from nfs. Are you positive your storage isn't having issues?

 

[james woods]

I typically hit this due to a storage issue from nfs. Are you positive your storage isn't having issues?

Hey, the storage is configured in the same way for each node, but only this one has the issue. Is there a quick way to check if NFS is causing it?

pvesm shows the storages as active.

 

[james woods]

It's also worth noting, that for the brief time I can get it showing when restarting pvestatd the containers still show up as question marks and pct list just hangs. qm list works fine and so does qm migrate. For now I'm migrating them off and then rebooting.

 

[adamb]

Hey, the storage is configured in the same way for each node, but only this one has the issue. Is there a quick way to check if NFS is causing it?

pvesm shows the storages as active.

If you click on the storage from the GUI, can you actually see whats inside?

 

[james woods]

If you click on the storage from the GUI, can you actually see whats inside?

I'm not too sure what you mean. If I go to Storage in the pane it lists my storages and if I double click it just brings up their settings.

If I log onto the node via SSH and go to the MNT folder, I can browse into the storages without any problems at all.

 

[adamb]

I'm not too sure what you mean. If I go to Storage in the pane it lists my storages and if I double click it just brings up their settings.

If I log onto the node via SSH and go to the MNT folder, I can browse into the storages without any problems at all.

From the GUI you should be able to highlight your storage and on the right side have tab's called Summary and Content. Within Content can you see your disks/backups?

 

[james woods]

From the GUI you should be able to highlight your storage and on the right side have tab's called Summary and Content. Within Content can you see your disks/backups?

Mine looks like this:

Clicking on the columns doesn't do anything besides highlight it.

 

[adamb]

Mine looks like this:
View attachment 12204
Clicking on the columns doesn't do anything besides highlight it.

This should help.

 

[james woods]

This should help.

View attachment 12208

Ah I see, the storages are also showing the same behaviour, they are just showing a question mark for node 1. Though I tried to browse to the content tab on another node and it does just say loading, which is strange, since the other 3 nodes are running perfectly fine and I can browse to them in the mnt folder via ssh.

 

[james woods]

Scratch that, I think I was just being impatient, it loaded fine after a little bit, besides on the affected node where it's just a question mark.

 

[adamb]

Scratch that, I think I was just being impatient, it loaded fine after a little bit, besides on the affected node where it's just a question mark.

It seems you have some type of storage issue on that specific node.

 

[james woods]

It seems you have some type of storage issue on that specific node.

I think it is something to do with the containers we are running. Since I can't do anything with them even from the cli. I've rebooted the node and migrated the containers to see if it makes the issue happen on another node. Thank you for your assistance.

 

[Chris]

I think it is something to do with the containers we are running. Since I can't do anything with them even from the cli. I've rebooted the node and migrated the containers to see if it makes the issue happen on another node. Thank you for your assistance.

Please check the syslog on the node with the issue, are there any errors around the time this happens?

 

[james woods]

Please check the syslog on the node with the issue, are there any errors around the time this happens?

Hello. While was ongoing there seems to be many entries in syslog saying pve-firewall[2473]: status update error: command '/sbin/iptables-save' failed: exit code 1
it seems to have happened every 10 seconds.

Towards the start of the issue happening there is also systemd[1]: Failed to propagate agent release messate: Transport endpoint is not connected

That one is in a block of 17 that all happened on the same second.

 

[Chris]

Maybe a firewall misconfiguration? Try to run pve-firewall compile on this host and check for issues.

 

[james woods]

That shows the following output:

 

[james woods]

Maybe a firewall misconfiguration? Try to run pve-firewall compile on this host and check for issues.

Sorry I didn't quote you, so I guess you probably won't get a notification. I posted the screenshot above

 

[Chris]

Well this excludes the firewall as possible issue since it is not enabled. What is the output of pvecm status when this happens?

 

[james woods]

Well this excludes the firewall as possible issue since it is not enabled. What is the output of pvecm status when this happens?

Hi there, I was waiting for the issue to happen again.

I have moved the containers we had to another node, and now it is that node having the problem. So it seems to be linked to the containers. The firewall is also disabled on this node.

Here is the output of pvecm status: