[SOLVED] Backup jobs fail after interrupted VM clone

[Chicken76]

Backup jobs have failed with the message:
ERROR: Backup of VM 104 failed - VM 104 qmp command 'backup' failed - Node 'drive-scsi1' is busy: block device is in use by block job: mirror

So I did the following:
qm monitor 104
qm> info block-jobs
Type mirror, device drive-scsi1: Completed 1126247104512 of 1126247104512 bytes, speed limit 0 bytes/s
Type mirror, device drive-scsi0: Completed 107454398464 of 107454398464 bytes, speed limit 0 bytes/s
qm> block_job_cancel drive-scsi1
Error: Job 'drive-scsi1' in state 'concluded' cannot accept command verb 'cancel'
qm> block_job_cancel drive-scsi0
Error: Job 'drive-scsi0' in state 'concluded' cannot accept command verb 'cancel'
qm> info block-jobs
Type mirror, device drive-scsi1: Completed 1126247104512 of 1126247104512 bytes, speed limit 0 bytes/s
Type mirror, device drive-scsi0: Completed 107454398464 of 107454398464 bytes, speed limit 0 bytes/s

Now the backups fail with:
ERROR: Backup of VM 104 failed - VM 104 qmp command 'backup' failed - backup_job_create failed: Job ID 'drive-scsi0' already in use


What happened:
I tried to clone this VM from the web interface, but it being a pretty big one, the VM started responding sporadically, so this being during work hours I hit the stop button, but this gave an error and the load on the server continued even if the job showed as ended. Having no other choice, I took the usability hit and eventually it calmed down. Then I deleted the partially done clone. Everything seemed to be working fine, except this morning I saw the nightly backup jobs for this VM failed.

So now I'm afraid to shut down this VM. I'm not going to lose the storage devices, right? /bitingnails

 

[fiona]

Hi,
the mirror jobs are concluded, so they were already canceled. You should be able to dismiss them:

echo '{"execute": "qmp_capabilities"}{"execute": "job-dismiss", "arguments": { "id": "drive-scsi0"}}{"execute": "job-dismiss", "arguments": { "id": "drive-scsi1"}}' | socat - /run/qemu-server/104.qmp

You should not lose the storage devices by shutting down.

The cleanup of the jobs probably couldn't be finished because stopping the task took longer than 5 seconds (at which time the task is killed): https://bugzilla.proxmox.com/show_bug.cgi?id=5471

 

[Chicken76]

Thank you @fiona

I bit the bullet and shut down the VM during lunch break. No problems starting it up after. Doing a manual backup right now and it's running normally.

So the problem is solved and the thread can be closed.