[SOLVED] Backup Job leads to complete Node Lockup

[Gamienator]

Hello everyone,

sorry for the small amount of details but at the moment I don't have any clue how to debug it.

For two Days my Proxmox Nodes lockes completly up when a backup Job is done. The only way to get my systems back running is with:

systemctl --force --force reboot

How should I start debugging this issues?

Cheers,
Gamie

 

[Moayad]

Hello,

You can check the syslog/journalctl to see that exactly happens when the backup is finish on the node, using journalctl you can sort the specific syslog on a specific time. e.g.:

journalctl --since "2023-05-01 00:00" --until "2023-05-01 08:45" > /tmp/Syslog.log

You have to change the time/date in the above command, as the backup time.

 

[Gamienator]

Thanks for the command:

I guess I found the reason why everything is breaking:

May 02 13:21:18 proxmox2 pvedaemon[2063469]: <root@pam> successful auth for user 'root@pam'
May 02 13:21:33 proxmox2 zed[2238072]: eid=30 class=checksum pool='rpool' vdev=nvme-eui.8ce38e100037bcf3-part3 algorithm=fletcher4 size=40960 offset=372362838016 priority=0 err=52 flags=0x180880 bookmark=34973:294370:0>
May 02 13:21:33 proxmox2 zed[2238070]: eid=29 class=data pool='rpool' priority=0 err=52 flags=0x808881 bookmark=34973:294370:0:0
May 02 13:21:33 proxmox2 zed[2238079]: eid=31 class=data pool='rpool' priority=0 err=52 flags=0x8881 bookmark=34973:241353:0:2
May 02 13:21:33 proxmox2 zed[2238081]: eid=32 class=checksum pool='rpool' vdev=nvme-eui.8ce38e100037bcf3-part3 algorithm=fletcher4 size=131072 offset=374686412800 priority=0 err=52 flags=0x180880 bookmark=34973:241353:>
May 02 13:21:33 proxmox2 zed[2238090]: eid=33 class=data pool='rpool' priority=0 err=52 flags=0x808881 bookmark=34973:241191:0:4
May 02 13:21:33 proxmox2 zed[2238092]: eid=34 class=checksum pool='rpool' vdev=nvme-eui.8ce38e100037bcf3-part3 algorithm=fletcher4 size=114688 offset=371454042112 priority=0 err=52 flags=0x180880 bookmark=34973:241191:>
May 02 13:21:33 proxmox2 zed[2238101]: eid=36 class=checksum pool='rpool' vdev=nvme-eui.8ce38e100037bcf3-part3 algorithm=fletcher4 size=114688 offset=378802225152 priority=0 err=52 flags=0x180880 bookmark=34973:241379:>
May 02 13:21:33 proxmox2 zed[2238099]: eid=35 class=data pool='rpool' priority=0 err=52 flags=0x808881 bookmark=34973:241379:0:1
May 02 13:21:33 proxmox2 zed[2238108]: eid=37 class=data pool='rpool' priority=0 err=52 flags=0x808881 bookmark=34973:241387:0:2

To bad that this drive is not mirrored But lets see what data I'm able to get back. I was in the middle of doing my backups