Backups started failing

[gob]

Hi

running PVE 2.3-13 I experienced a virt server running really slowly so I shut down both guest servers and rebooted the host server.
Since then, the nightly backups (to USB disk) have failed with:

[COLOR=#000000][FONT=tahoma]INFO: starting new backup job: vzdump 100 101 --quiet 1 --mailto xxx@xxx.com --mode snapshot --compress lzo --storage USB-Drive --node prox1[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 100 (qemu)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: VM is locked (backup)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]ERROR: Backup of VM 100 failed - command 'qm set 100 --lock backup' failed: exit code 25[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 101 (qemu)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: unable to open file '/etc/pve/nodes/prox1/qemu-server/101.conf.tmp.83789' - Input/output error[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]ERROR: Backup of VM 101 failed - command 'qm set 101 --lock backup' failed: exit code 2[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Backup job finished with errors[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK ERROR: job errors[/FONT][/COLOR]

I assumed the VM's were locked so I issued the usual:

qm unlock 101

but this now results in:

unable to open file '/etc/pve/nodes/prox1/qemu-server/101.conf.tmp.84865' - Input/Output error

Any suggestions?

thanks
Gordon

 

[dietmar]

You run a cluster, and the cluster is not quorate?

 

[gob]

No cluster. A standalone host with local storage.

 

[dietmar]

Is the pve-cluster service running (you need that even for non-clusterd installations)? If not, try to restart

# service pve-cluster restart

 

[gob]

Thanks Dietmar
I couldn't see the cluster service running with ps -aux, however issuing a service stop and start didn't report that it wasn't running so perhaps it was.
Since restarting if I issue a 'qm unlock 100' I no longer get an error message so perhaps that has sorted the problem.
I will know more tomorrow when the backup attempts to run again.

Thanks
Gordon

 

[dietmar]

You can easily check if the pve cluster service (pmxcfs) runs correctly - it is the fuse file system mounted at /etc/pve. So you can simple do:

# ls -l /etc/pve

to check if it works.