systemd: failed to start Journal Service

[Alessandro 123]

Hi to all.
It's the fourth time in 3 days that a PVE 5 box suddenly crash (maybe on heavy I/O) with the following errors visible on screen:

systemd[1]: Failed to start Journal Service
systemd[1]: Failed to start udev Kernel Device Manager

Any clue?
ping is working, everything else not.

 

[Alwin]

You don't have access to the console? There maybe some more information.

 

[Alessandro 123]

No, when this happens the server is totally gone.
Log files doesn't log anything useful.

I asked a colleague of mine to take a photo at screen just before hard reboot and it was full of:

task kswapd0:81 blocked for more than 120 seconds
        Tainted: P              O       4.10.17-2-pve #1

task txg_sync:1999 blocked for more than 120 seconds
        Tainted: P              O       4.10.17-2-pve #1

and so on.

 

[Alwin]

Check your zfs settings on you host, as you mentioned high I/O. The server might just swap out and stopp being responsive.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_zfs_on_linux

 

[Alessandro 123]

Damn! totally forgot to lower down the swappiness.

Now I'm trying "10", but probably, i'll go with "1"

 

[Alessandro 123]

This night server rebooted automatically, maybe due to swappiness set to 10?
In fact, I can see a very low swap usage, but I don't know why server rebooted automatically.

I don't see anything in any log files, last wrote log line was as expected (the output of rsnapshot), immediatly after i can only see the system boot up.