Proxmox crashing with "Journal has aborted" message

[blueshift]

Hello, I am using a Protectli VP3230 with a PCI-E card and also a default Protectli NVMe-SATA-adapter (with one SATA-SSD connected) on both NVMe-connectors. A few months ago my Proxmox suddenly crashed with an error message "EXT4-fs error (device dm-1) in ext4_reserve_fnode_write:5792: Journal has aborted"!

First time seeing that I thought it was a filesystem error, so I checked that and I could repair it. Then it happened again but all was fine. Then I updated Proxmox to version 8 but it happened again. Then I replaced the SSD, re-installed Proxmox 8 and configured it from scratch - and only imported my vms and also checked the RAM.

Since that, all was fine. Until today... same error message, same problem!

What else could cause those errors? They start to be annoying because my full system goes down and I have to manually interact to restart it again...

Thanks for your help, hints and advices!

 

[Kingneutron]

In all likelihood, the make/model of disk you are using is not good with proxmox. Consumer- and desktop-rated drives are widely known to be a bad mix with a 24/7 hypervisor.

https://protectli.com/product/vp3230/

You have 2x m.2 nvme slots, and 4x 2.5-inch SSD connectors, why are you using an adapter?

Post details of make/model/size of disks, and I would also recommend putting everything on UPS power if you haven't already.

 

[blueshift]

Hello King,

I am also using another Protectli for years with Proxmox and that works perfectly fine (also without enterprise hardware). The VP3230 got only two NVMe-connectors and 4 SATA-power connectors... so you can connect 4 SATA SSDs using the Protectli NVMe-SATA-adapter

You can see details here:

https://kb.protectli.com/kb/vp3200-series-hardware-overview/

I am already using a UPS, to be sure it is all secure. And actually I got a Samsung 860 EVO V-NAND 1TB built in (the SSDs are totally fine, I checked all... filesystem, SMART, etc.).

 

[Kingneutron]

You could try extending the write delay, add " ,commit=20 " to /etc/fstab options for rootfs (after the remount-ro), reboot and check dmesg and logs for errors. Changing the commit will delay writes for 20 seconds, allowing more aggregation.

If it keeps happening, you might want to reinstall with ZFS boot/root but be aware that the PVE installer will wipe the target disk(s) so make sure you backup everything - preferably to PBS on separate hardware

https://github.com/kneutron/ansitest/tree/master/proxmox

Look into the bkpcrit script, point it to external disk / NAS, run it nightly in cron