Proxmox node is offline but everything else is running fine

D

devkid

New Member
Apr 12, 2024
5
0
1
Hi,

Please can I get some help. As the title implies, I have a proxmox node that goes offline, but the containers and VMs are still running fine.
I can ssh to the proxmox node but I'm not sure what to check.
The only thing I have checked is the disk space and that is fine also.

I also tried systemctl restart pveproxy but that seems like it hangs for quite a while and then it displays this:
Code: 
Job for pveproxy.service failed because a timeout was exceeded.
See "systemctl status pveproxy.service" and "journalctl -xeu pveproxy.service" for details.

i tried journalctl -xeu pveproxy.service and it shows me this:

Code: 
░░ Subject: A stop job for unit pveproxy.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A stop job for unit pveproxy.service has finished.
░░
░░ The job identifier is 2397 and the job result is done.
Oct 03 08:38:30 SonyProx systemd[1]: pveproxy.service: Found left-over process 1714 (pveproxy) in control group while starting unit. Ignoring.
Oct 03 08:38:30 SonyProx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 03 08:38:30 SonyProx systemd[1]: pveproxy.service: Found left-over process 69124 (pveproxy worker) in control group while starting unit. Ignoring.
Oct 03 08:38:30 SonyProx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 03 08:38:30 SonyProx systemd[1]: pveproxy.service: Found left-over process 74664 (pveproxy worker) in control group while starting unit. Ignoring.
Oct 03 08:38:30 SonyProx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 03 08:38:30 SonyProx systemd[1]: pveproxy.service: Found left-over process 77089 (pveproxy worker) in control group while starting unit. Ignoring.
Oct 03 08:38:30 SonyProx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 03 08:38:30 SonyProx systemd[1]: pveproxy.service: Found left-over process 81379 (pveproxy) in control group while starting unit. Ignoring.
Oct 03 08:38:30 SonyProx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 03 08:38:30 SonyProx systemd[1]: pveproxy.service: Found left-over process 139724 (pveproxy) in control group while starting unit. Ignoring.
Oct 03 08:38:30 SonyProx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 03 08:38:30 SonyProx systemd[1]: pveproxy.service: Found left-over process 140609 (pvecm) in control group while starting unit. Ignoring.
Oct 03 08:38:30 SonyProx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 03 08:38:30 SonyProx systemd[1]: pveproxy.service: Found left-over process 141551 (pvecm) in control group while starting unit. Ignoring.
Oct 03 08:38:30 SonyProx systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 03 08:38:30 SonyProx systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
░░ Subject: A start job for unit pveproxy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pveproxy.service has begun execution.
░░
░░ The job identifier is 2397.
Oct 03 08:40:00 SonyProx systemd[1]: pveproxy.service: start-pre operation timed out. Terminating.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: State 'stop-sigterm' timed out. Killing.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 142437 (pvecm) with signal SIGKILL.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 1714 (pveproxy) with signal SIGKILL.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 69124 (pveproxy worker) with signal SIGKILL.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 74664 (pveproxy worker) with signal SIGKILL.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 77089 (pveproxy worker) with signal SIGKILL.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 81379 (pveproxy) with signal SIGKILL.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 139724 (pveproxy) with signal SIGKILL.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 140609 (pvecm) with signal SIGKILL.
Oct 03 08:41:30 SonyProx systemd[1]: pveproxy.service: Killing process 141551 (pvecm) with signal SIGKILL.
Oct 03 08:43:00 SonyProx systemd[1]: pveproxy.service: Processes still around after SIGKILL. Ignoring.

does anyone have any ideas?
 
I have checked the log via journalctl -b and it shows the below around the time the node had a problem.

Code: 
Oct 02 23:59:48 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:00:03 SonyProx pveproxy[77089]: proxy detected vanished client connection
Oct 03 00:00:03 SonyProx systemd[1]: Starting dpkg-db-backup.service - Daily dpkg database backup service...
Oct 03 00:00:03 SonyProx systemd[1]: Starting logrotate.service - Rotate log files...
Oct 03 00:00:03 SonyProx systemd[1]: dpkg-db-backup.service: Deactivated successfully.
Oct 03 00:00:03 SonyProx systemd[1]: Finished dpkg-db-backup.service - Daily dpkg database backup service.
Oct 03 00:00:03 SonyProx systemd[1]: Reloading pveproxy.service - PVE API Proxy Server...
Oct 03 00:00:13 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:00:23 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:00:24 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:00:33 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:00:42 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:00:43 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:00:53 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:00:53 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:00:03 SonyProx systemd[1]: Finished dpkg-db-backup.service - Daily dpkg database backup service.
Oct 03 00:00:03 SonyProx systemd[1]: Reloading pveproxy.service - PVE API Proxy Server...
Oct 03 00:00:13 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:00:23 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:00:24 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:00:33 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:00:42 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:00:43 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:00:53 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:00:53 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:01:33 SonyProx systemd[1]: pveproxy.service: Reload operation timed out. Killing reload process.
Oct 03 00:01:33 SonyProx systemd[1]: Reload failed for pveproxy.service - PVE API Proxy Server.
Oct 03 00:01:33 SonyProx systemctl[81378]: Job for pveproxy.service failed.
Oct 03 00:01:33 SonyProx systemctl[81378]: See "systemctl status pveproxy.service" and "journalctl -xeu pveproxy.service" for details.
Oct 03 00:01:33 SonyProx systemd[1]: Reloading spiceproxy.service - PVE SPICE Proxy Server...
Oct 03 00:01:47 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:01:58 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:02:32 SonyProx pmxcfs[1542]: [dcdb] notice: data verification successful
Oct 03 00:02:34 SonyProx pveproxy[77089]: proxy detected vanished client connection
Oct 03 00:02:52 SonyProx pveproxy[77089]: proxy detected vanished client connection
Oct 03 00:03:03 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:03:03 SonyProx systemd[1]: spiceproxy.service: Reload operation timed out. Killing reload process.
Oct 03 00:03:03 SonyProx systemd[1]: Reload failed for spiceproxy.service - PVE SPICE Proxy Server.
Oct 03 00:03:03 SonyProx systemctl[81560]: Job for spiceproxy.service failed.
Oct 03 00:03:03 SonyProx systemctl[81560]: See "systemctl status spiceproxy.service" and "journalctl -xeu spiceproxy.service" for details.
Oct 03 00:03:03 SonyProx logrotate[81306]: error: error running shared postrotate script for '/var/log/pveproxy/access.log '
Oct 03 00:03:03 SonyProx systemd[1]: Stopping pvefw-logger.service - Proxmox VE firewall logger...
Oct 03 00:03:03 SonyProx pvefw-logger[1233]: received terminate request (signal)
Oct 03 00:03:03 SonyProx pvefw-logger[1233]: stopping pvefw logger
Oct 03 00:03:04 SonyProx systemd[1]: pvefw-logger.service: Deactivated successfully.
Oct 03 00:03:04 SonyProx systemd[1]: Stopped pvefw-logger.service - Proxmox VE firewall logger.
Oct 03 00:03:04 SonyProx systemd[1]: pvefw-logger.service: Consumed 1.561s CPU time.
Oct 03 00:03:04 SonyProx systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
Oct 03 00:03:04 SonyProx pvefw-logger[81726]: starting pvefw logger
Oct 03 00:03:04 SonyProx systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
Oct 03 00:03:04 SonyProx systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Oct 03 00:03:04 SonyProx systemd[1]: logrotate.service: Failed with result 'exit-code'.
Oct 03 00:03:04 SonyProx systemd[1]: Failed to start logrotate.service - Rotate log files.
Oct 03 00:04:08 SonyProx pveproxy[77089]: proxy detected vanished client connection
Oct 03 00:04:33 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:04:43 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:04:53 SonyProx pveproxy[69124]: proxy detected vanished client connection
Oct 03 00:05:02 SonyProx pveproxy[74664]: proxy detected vanished client connection
Oct 03 00:05:03 SonyProx pveproxy[69124]: proxy detected vanished client connection
 
You must log in or register to reply here.
Top Bottom