Node's / filled now can't manage the node through the cluster.

[malvane]

Proxmox Version: 6.3-2 Four node cluster

Issue: A node's root (/) filled to 100%. Space was cleared but now when viewing the cluster through the WebUI the node and all of the VMs show as question marks.
pvecm status shows no issues.

Through SSH qm list hangs.

Nothing jumps out as being hung or not running. All VMs are still responding but I can't mange them.

Is there any further investigation to see what might be causing this to avoid a reboot of the node and all of the VMs under it. Or at least get it to the point where I can migrate the VMs to another node?

Thanks.

Edit: When attempting to directly login to the affected node through the WebUI I get the error Connection failure. Network error or Proxmox VE services not running?

 

[fiona]

Hi,
what's the output of systemctl status pveproxy.service pvedaemon.service pvestatd.service? You might also want to check /var/log/syslog for errors. Best to check on the problematic node and the node from which you access the web interface.

P.S. Proxmox VE 6.x is end-of-life. See here for how to upgrade.

 

[malvane]

Hi,
what's the output of systemctl status pveproxy.service pvedaemon.service pvestatd.service? You might also want to check /var/log/syslog for errors. Best to check on the problematic node and the node from which you access the web interface.

P.S. Proxmox VE 6.x is end-of-life. See here for how to upgrade.

Thanks for the info.

I am seeing some errors in syslog from pveproxy.

Aug 18 00:01:30 $SYSTEM systemd[1]: pveproxy.service: Reload operation timed out. Killing reload process.
Aug 18 00:01:30 $SYSTEM logrotate[11753]: Job for pveproxy.service failed.
Aug 18 00:01:30 $SYSTEM logrotate[11753]: See "systemctl status pveproxy.service" and "journalctl -xe" for details.

(slightly redacted)

Are the pveproxy, pvedaemon and pvestatd services safe to restart on the fly?

Thanks.

p.s. Yes, we have a project to move to version 7 but I'm the only sysadmin so it has been slow.

 

[Neobin]

If you want help, you should provide the complete output for all the commands you got asked for, especially if the Proxmox-Staff asked for it.
On top of this, in the only three output-lines you posted, it states to have a look (and since you ask for help here -> post the output here) into (partly) exactly this what fiona already asked for. But you still do not provide any output...

Not meant to be rude; only a friendly info and my 2 cents.

 

[fiona]

Aug 18 00:01:30 $SYSTEM systemd[1]: pveproxy.service: Reload operation timed out. Killing reload process.
Aug 18 00:01:30 $SYSTEM logrotate[11753]: Job for pveproxy.service failed.
Aug 18 00:01:30 $SYSTEM logrotate[11753]: See "systemctl status pveproxy.service" and "journalctl -xe" for details.

(slightly redacted)

It's also asking for the systemctl status output

Are the pveproxy, pvedaemon and pvestatd services safe to restart on the fly?

Yes, but it's a good idea to find out why the timeout happened first. Otherwise it might just happen again.