Proxmox 8.0.3 systemd-journald.service It gives an error and the system stops

[kesen]

Hello Friends, I am very new to Proxmox systems. I installed Proxmox 8.0.3 version on HP DL380 GEN-9 Server. With ZFS Raid. and I added 1 storage as ZFS2 Raid. Everything went well until now (1 week). But now something is wrong. I see the error appearing below and it's pages long. The server stops responding and of course the VMs are down. Proxmox cannot be accessed via web or SSH. It gets better after restarting the server, but it reoccurs after a few hours.
I have no idea what I can check or where the problem might be coming from.

Since I thought everything was going well, I ran the VMs I created live and I'm having problems.




systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
systemd[1]: systemd-journald.service: Unit process 167**** (systemd-journal) remains running after unit stopped.
.....
.....
.....

systemd[1]: failed to start systemd-journald.service - journal Service.
systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 176.
systemd[1]: Stopped systemd-journald.service - Journal Service.

systemd[1]: systemd-journald.service: Found left-over process 1993**** (systemd-journal) in control group wile starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies
systemd[1]: systemd-journald.service: Found left-over process 1993**** (systemd-journal) in control group wile starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies
systemd[1]: systemd-journald.service: Found left-over process 1993**** (systemd-journal) in control group wile starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies
systemd[1]: systemd-journald.service: Found left-over process 1993**** (systemd-journal) in control group wile starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies
systemd[1]: systemd-journald[1109665]: File /var/log/journal/xxxxxxxxxxxxxxxxxx/system.journal corrupted or uncleanly shut down, renaming and replacing.
systemd[1]: systemd-journald.service: start operation timed out. Terminating.
.....
.....
.....

systemd[1]: systemd-journald.service: Killing process 167**** (systemd-journal) with signal SIGKILL.
systemd[1]: systemd-journald.service: Killing process 167**** (systemd-journal) with signal SIGKILL.
systemd[1]: systemd-journald.service: Killing process 167**** (systemd-journal) with signal SIGKILL.
systemd[1]: systemd-journald.service: Killing process 167**** (systemd-journal) with signal SIGKILL.
systemd[1]: systemd-journald.service: Killing process 167**** (systemd-journal) with signal SIGKILL.
systemd[1]: systemd-journald.service: Killing process 167**** (systemd-journal) with signal SIGKILL.
systemd[1]: systemd-journald.service: Killing process 167**** (systemd-journal) with signal SIGKILL.
systemd[1]: systemd-journald.service: Killing process 167**** (systemd-journal) with signal SIGKILL.
.....
.....
.....

systemd[1]: systemd-journald.service: Processes still aruund after SIGKILL. Ignoring.

 

[LnxBil]

Sadly, none of the error messages is helpful in order to determine what went wrong. Can you try to post the output of dmesg (please in code tags)

 

[kesen]

Thanks for the reply, I just saw what you wrote. I'm looking for a solution.
The entire code did not fit on the page. The entire thing is available in the attachment.

(I revised the attached dmesg file.)

 

[kesen]

Hello,
I'm having the same problem again. If downgrade ProxMox to version 7 will it be more stable?
I don't know if I can switch from V8 to V7. I don't know if it will be of any use?

Is there anyone with knowledge who can guide me about this error and what I can do?

 

[LnxBil]

The entire thing is available in the attachment.

Unfortunately, nothing in there that can help us.

Is there anyone with knowledge who can guide me about this error and what I can do?

We have still not seen any error message. What you see on your screen is smoke, we need to find the fire.

 

[chedda7]

Seeing similar errors but like @LnxBil says, can't find any bonafide errors.

My thread: https://forum.proxmox.com/threads/proxmox-ve-8-cluster-new-nodes-unhealthy.135087/

We did convert one cluster node to non-RAID and it appears to be more stable.

 

[kesen]

Hello,
Thank you for trying to help and thank you for the answers.When I turned off all backup schedules, the problem seemed to be resolved.
I haven't experienced any system fault in 24 hours. I will examine it, but I cannot be at peace without making a backup.

 

[muekno]

Same problem I am on 8.1.10, running since about two weeks everythings very fine, but no since about two hours I/O delay goes up to about 50% everthing is extrem slow, trying to shut down VMs, via console, does work, "shutdown -h now" via ssh on debian 12 VM, cut the ssh connection but takes long time until no more pingable
PVE Console shows still running.

I am not realy able to reboot everything