Backup failed due to busy device

[oldgoodname]

Hey Guys,

I have opened a thread about the same problem some months ago: https://forum.proxmox.com/threads/backup-error-over-smb-cifs-from-pve.140354/#post-633459
I thought it's better to open a new one, hope thats OK.

Here is a short overview of my network:

• 2x PVE host in a cluster
• 1x Quorum virtual machine running debian and hosted on the Qnap NAS
• 1x Qnap NAS with 2 storage pools
  • 1 HDD storage pool saving backup-data
  • 1 SSD storage pool running all virtual machines (quorum and pve virtual machines)
• 1x PBS host virtualized on one of the PVE hosts backing up data to a NFS share on the HDD storage of the Qnap NAs

The PVE hosts, quorum vm and the Qnap NAS are on the same vlan, only the PBS is on a different one routed over a firewall.


Since the beginning I have problems with backing up my virtual machines. For some months I haven't done any backups and was thinking about using a third party product or waiting for veeam for proxmox. About a month ago, I started a new try with the PBS to backup my vms as I upgraded from PVE version 7 to 8. It worked for about a month backuping up one cluster node a 1am and the other cluster node at 4am.

Yesterday I wanted to create some manual backups, but they failed several times. While the primary node created a successfull backup of the vms, the second node wasn't able to do so for several times, even after restarting node 1 and node 2. It always failed after the first or the second vm. So I forgot about the backup an took the risk. Today I hoped that it would create a successful backup again at 4am, but it failed. That have worked for about a month.

As the backup is really essential now, I need to get that fixed. Hopefully you can help me with that. One of the VMs always remain in a locked state (103 in the logfile), which I have to manually unlock on the terminal of the host. I attached the log file of today. The only thing I deleted from the logfile are about 20 - 30 entries of a service user that logs in. This is form the monitoring system and I think not relevant for the case.

For me it looks like it loses quorum status from time to time, but how can I fix that? Is there an option to prioritize this kind of traffic?

I really appreciate your help.
Thanks and best regards.

 

[Johannes S]

Why didn't you setup pbs together with the qdevice vm on the qnap? I would expect that you will get at least better Performance than with your current setup since local disks are the only supported storage for PBS.

 

[oldgoodname]

I wanted to get rid of all VMs on the QNap as I want to exchange it sometimes with truenas or similar. So I installed everything on the PVE even I know a standalone PBS would make more sense. Later I thought the only good way to implement a quorum is to install a VM on the Qnap as installing it on the PVE would make no sense. So I used the Qnap again as a hypervisor.

I can install a new PBS on the Qnap, but this would be on the SSD storage pool, while the data would be backed up onto the HDD storage pool. I don't know if Qnap supports that, to directly mount another storage pool or if I have to use the fallback of NFS again. I first have to check that. But then I still don't know if the backup would be directly transfered from the SSD pool to the HDD pool. I think it wil always communicate and copy the data through the network. So the performance may be the same.

But the performance itself isn't the problem. The backup of one PVE node just needs 15 minutes, but I am wondering, why the specific error occurs.

 

[fabian]

that looks like your backup traffic/load either overloads the network, or the qnap box entirely.. I don't think there is a good way out of this other than spec-ing your hardware accordingly. not sure whether your "quorum VM" is a full blown PVE, or just a qdevice, if it's the former, switching to a qdevice might make quorum more stable.