Backup issues lockup server, what's the most reliable backup method?

[Kevo]

Proxmox has been quite reliable for me but I have had issues from time to time. It's seems the most common issue I have is a backup failing for some reason. In some cases the backup fails because of a drive problem, and in other cases the issue is unknown. My latest failure I reboot and rerun the backup and it's all fine. It could be that the drive is failing and just doesn't show any detectable signs in SMART or with typical testing. Regardless, I would like to find a way to make proxmox more robust in this scenario. Ideally it would report an error with the backup and just keep working, but that hasn't been my experience.

My usual backup method is to install a standalone drive and setup a directory for backups to be written to using the built in backup function. I've seen the backup server but have never used it as I just have one server to run all my test vms and such on. Would running the backup server mitigate issues with the built in proxmox backup failing causing server lockups or is there some other backup method I should be looking at to make the server more robust. TIA

 

[sb-jw]

PVE itself is very robust and works very well. For this to be the case, however, it is of course necessary that the hardware used works flawlessly or is replaced promptly.

Can you give us more details about your system and bugs (e.g. log entries)?

 

[Kevo]

It's usually the result of a drive failing or beginning to fail. In this latest case there is an IO failure on the backup drive that was logged. SMART shows OK though and rerunning the backups worked fine. I suspect that the drive will flat out fail in the next few weeks or months so I am going to get a new one and replace it.

I guess I don't fully understand why an IO failure of a drive used for backup takes down the whole system. Maybe it's a linux issue that Proxmox can't avoid? Do you know if running the backup server on a separate machine would resolve this type of failure? Maybe I should look into putting an old machine into service just for backups.

 

[leesteken]

I guess I don't fully understand why an IO failure of a drive used for backup takes down the whole system.

That is weird and not normal. Any clues in journalctl around that time? Or maybe on a physical display connected to the server (if it could not flush the logs)?

 

[Kevo]

IIRC the error on the screen at the time was systemd-journald.service failed to start or something like that. I'll have to check logs more thoroughly when I have some time and report back.