Failed backup on possible locked VM

[Andrew Holybee]

I had a failed backup happen Monday where I had to restart the NFS that froze
Tuesday 1 vm worked and the other vm failed here is the out put it looks like its still locked.
104: Jul 11 02:13:47 INFO: Starting Backup of VM 104 (qemu)
104: Jul 11 02:13:47 INFO: status = running
104: Jul 11 02:13:48 INFO: VM is locked (backup)
104: Jul 11 02:13:48 ERROR: Backup of VM 104 failed - command 'qm set 104 --lock backup' failed: exit code 255

My question is can I just do "qm unlock 104" so it will be able to back up tonight and will it affect the VM while its running?

 

[dietmar]

My question is can I just do "qm unlock 104" so it will be able to back up tonight and will it affect the VM while its running?

You need to make sure that the previous backup job does not run anymore. Then you can unlock (it will not affect the VM while its running).

 

[Andrew Holybee]

Looks like it stopped as near as I can tell here a screenshot. Do I need to execute a command to check like
"# egrep '^lock: backup$' /etc/qemu-server/104.conf;"

 

[dietmar]

Yes, seems to be stopped. Please can you post the backup log - I wonder what error left the VM in locked state.

 

[Andrew Holybee]

looks like my NAS became unresponsive. I clicked on the stop to stop the backup and then I had to restart the NAS itself
Jul 10 05:33:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:33:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:33:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:34:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:34:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:34:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:34:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:34:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:34:53 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:35:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:35:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:35:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:35:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:35:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:35:53 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:36:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:36:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:36:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:36:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:36:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:36:53 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:37:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:37:13 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:37:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:37:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:37:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:37:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:38:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:38:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:38:17 pm02 pmxcfs[1911]: [status] notice: received log
Jul 10 05:38:17 pm02 sshd[17565]: Accepted publickey for root from 192.168.1.xx port 57672 ssh2: RSA
Jul 10 05:38:17 pm02 sshd[17565]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 10 05:38:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:38:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:38:34 pm02 sshd[17565]: Received disconnect from 192.168.1.xx: 11: disconnected by user
Jul 10 05:38:34 pm02 sshd[17565]: pam_unix(sshd:session): session closed for user root
Jul 10 05:38:34 pm02 pmxcfs[1911]: [status] notice: received log
Jul 10 05:38:35 pm02 sshd[17616]: Accepted publickey for root from 192.168.1.xx port xxxxx ssh2: RSA xxx
Jul 10 05:38:35 pm02 sshd[17616]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 10 05:38:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:38:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:39:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:39:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:39:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:39:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:39:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:39:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:40:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:40:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:40:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:40:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:40:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:40:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:41:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:41:13 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:41:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:41:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:41:39 pm02 sshd[17616]: Received disconnect from 192.168.1.xx: 11: disconnected by user
Jul 10 05:41:39 pm02 sshd[17616]: pam_unix(sshd:session): session closed for user root
Jul 10 05:41:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:41:53 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:42:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:42:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:42:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:42:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:42:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:42:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:43:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:43:13 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:43:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:43:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:43:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:43:53 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:44:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:44:13 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:44:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:44:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:44:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:44:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:45:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:45:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:45:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:45:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:45:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:45:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:46:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:46:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:46:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:46:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:46:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:46:53 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:47:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:47:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:47:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:47:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:47:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:47:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:48:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:48:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:48:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:48:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:48:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:48:53 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:49:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:49:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:49:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:49:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:49:33 pm02 smartd[1758]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 70 to 69
Jul 10 05:49:33 pm02 smartd[1758]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 30 to 31
Jul 10 05:49:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:49:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:50:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:50:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:50:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:50:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:50:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:50:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:51:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:51:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:51:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:51:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:51:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:51:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:52:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:52:13 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:52:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:52:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:52:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:52:53 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:53:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:53:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:53:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:53:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:53:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:53:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:54:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:54:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:54:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:54:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:54:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:54:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:55:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:55:13 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:55:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:55:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:55:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:55:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:56:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:56:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:56:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:56:23 pm02 pmxcfs[1911]: [status] notice: received log
Jul 10 05:56:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:56:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:56:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:57:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:57:13 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:57:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:57:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:57:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:57:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:58:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:58:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:58:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:58:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:58:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:58:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:59:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:59:13 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:59:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:59:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:59:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 05:59:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:00:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:00:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:00:22 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:00:33 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:00:42 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:00:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:01:02 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:01:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:01:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:01:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:01:43 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:01:52 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:02:03 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:02:12 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:02:23 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:02:32 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:02:40 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:02:50 pm02 pvestatd[2277]: storage 'Backups' is not online
Jul 10 06:03:21 pm02 systemd-timesyncd[1356]: interval/delta/delay/jitter/drift 2048s/-0.005s/0.034s/0.007s/+45ppm
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:53 pm02 kernel: nfs: server 192.168.1.23 OK
Jul 10 06:04:54 pm02 pvestatd[2277]: status update time (113.524 seconds)
Jul 10 06:04:58 pm02 vzdump[24562]: <root@pam> end task UPIDm02:00005FF3:04BC3D8D:596325F4:vzdump::root@pam: unexpected status
Jul 10 06:04:58 pm02 CRON[24560]: pam_unix(cron:session): session closed for user root
Jul 10 06:05:31 pm02 pmxcfs[1911]: [status] notice: received log

 

[Andrew Holybee]

ok so opened shell on correct node now and gave unlocked command it didn't say anything so i guess i will see if that fixed it.

 

[Andrew Holybee]

yep it fixed it backed up just fine