TASK ERROR: verification failed, no further information

[carsten2]

After running for 1,5 days my verification job failed with error "TASK ERROR: verification failed - please check the log for details", but the whole log only says "0 errors" for each step. Where to find information, what failed exactly?

BTW: It would be a good thing, if it were possible to download a a task log from the UI, so search in it.

 

[fabian]

did you maybe stop the task (e.g. by stopping the proxmox-backup-proxy)?

 

[carsten2]

no. Could it happen automatically? Anyway: the log should tell more about the error.

 

[fabian]

could you post the last lines of the log, and the status view?

 

[carsten2]

Here the log:

2020-11-16T06:49:17+01:00: verify pvebackup:vm/145/2020-09-08T01:21:03Z
2020-11-16T06:49:17+01:00: check qemu-server.conf.blob
2020-11-16T06:49:17+01:00: check drive-ide0.img.fidx
2020-11-16T06:49:17+01:00: verified 0.00/0.00 MiB in 0.00 seconds, speed 0.00/0.00 MiB/s (0 errors)
2020-11-16T06:49:17+01:00: percentage done: 100.91% (1104 of 1094 snapshots)
2020-11-16T06:49:17+01:00: verify pvebackup:vm/145/2020-09-07T02:17:09Z
2020-11-16T06:49:17+01:00: check qemu-server.conf.blob
2020-11-16T06:49:17+01:00: check drive-ide0.img.fidx
2020-11-16T06:49:18+01:00: verified 0.00/0.00 MiB in 0.00 seconds, speed 0.00/0.00 MiB/s (0 errors)
2020-11-16T06:49:18+01:00: percentage done: 101.01% (1105 of 1094 snapshots)
2020-11-16T06:49:18+01:00: verify pvebackup:vm/145/2020-09-06T14:37:17Z
2020-11-16T06:49:18+01:00: check qemu-server.conf.blob
2020-11-16T06:49:18+01:00: check drive-ide0.img.fidx
2020-11-16T06:49:18+01:00: verified 0.00/0.00 MiB in 0.00 seconds, speed 0.00/0.00 MiB/s (0 errors)
2020-11-16T06:49:18+01:00: percentage done: 101.10% (1106 of 1094 snapshots)
2020-11-16T06:49:18+01:00: verify group pvebackup:vm/208
2020-11-16T06:49:18+01:00: TASK ERROR: verification failed - please check the log for details

BTW: It would be enough the just verify only 100%

status line:

 

[fabian]

btw you can get the task log with proxmox-backup-manager task log <UPID>, if possible could you post it here? alternatively, if it is a simple 'verify everything' job you can run it with the 'Verify All' button which should print a nice summary of the failed things at the end.. I'll try to get this into the scheduled job log as well..

 

[carsten2]

Where to find the task id? I tried the following:

# proxmox-backup-manager task log pvebackup:v-bdc1352d-dd0b
Error: unable to parse UPID 'pvebackup:v-bdc1352d-dd0b'
# proxmox-backup-manager task log v-bdc1352d-dd0b
Error: unable to parse UPID 'v-bdc1352d-dd0b'
proxmox-backup-manager task log 'pvebackup:v-bdc1352d-dd0b'
Error: unable to parse UPID 'pvebackup:v-bdc1352d-dd0b'

I now started a verification with the button and will report.

 

[fabian]

if you double click the task entry (which opens the log in a dialogue), there is a second tab in the window that says 'Status', there you have the 'Unique Task ID'

 

[carsten2]

The new manual verification ends with the following error:

2020-11-18T15:31:32+01:00: verify pvebackup:vm/145/2020-09-06T14:37:17Z
2020-11-18T15:31:32+01:00: check qemu-server.conf.blob
2020-11-18T15:31:32+01:00: check drive-ide0.img.fidx
2020-11-18T15:31:32+01:00: verified 0.00/0.00 MiB in 0.00 seconds, speed 0.00/0.00 MiB/s (0 errors)
2020-11-18T15:31:32+01:00: percentage done: 100.70% (1146 of 1138 snapshots)
2020-11-18T15:31:32+01:00: verify group pvebackup:vm/208
2020-11-18T15:31:32+01:00: Failed to verify following snapshots/groups:
2020-11-18T15:31:32+01:00: ct/103/2020-10-03T22:14:06Z
2020-11-18T15:31:32+01:00: TASK ERROR: verification failed - please check the log for details

The interesting think is that vm/208 is not listed in contents.

How to fix this?

I am still not able to get the log for the job. The unique task id is
UPID:hal9034:0000062E:00000427:00000090:5FB21308:verificationjobvebackup\x3av\x2dbdc1352d\x2ddd0b:root@pam:
but the command:

proxmox-backup-manager task log "UPID:hal9034:0000062E:00000427:00000090:5FB21308:verificationjobvebackup\x3av\x2dbdc1352d\x2ddd0b:root@pam:"

just gives "Error:". No matter if I use it with or without quotes.

 

[dcsapak]

t

just gives "Error:". No matter if I use it with or without quotes.

thanks, with this i found a bug in proxmox-backup-manager. i sent a patch to the list https://lists.proxmox.com/pipermail/pbs-devel/2020-November/001559.html

for now you can open the web ui -> administration -> tasks and double click on the task and copy the content

 

[carsten2]

1) How to fix the verification error?
2) How to remove vm/208, which is not listed in contents?

 

[fabian]

The new manual verification ends with the following error:

2020-11-18T15:31:32+01:00: verify pvebackup:vm/145/2020-09-06T14:37:17Z
2020-11-18T15:31:32+01:00: check qemu-server.conf.blob
2020-11-18T15:31:32+01:00: check drive-ide0.img.fidx
2020-11-18T15:31:32+01:00: verified 0.00/0.00 MiB in 0.00 seconds, speed 0.00/0.00 MiB/s (0 errors)
2020-11-18T15:31:32+01:00: percentage done: 100.70% (1146 of 1138 snapshots)
2020-11-18T15:31:32+01:00: verify group pvebackup:vm/208
2020-11-18T15:31:32+01:00: Failed to verify following snapshots/groups:
2020-11-18T15:31:32+01:00: ct/103/2020-10-03T22:14:06Z
2020-11-18T15:31:32+01:00: TASK ERROR: verification failed - please check the log for details

The interesting think is that vm/208 is not listed in contents.

How to fix this?

the log says that 'ct/103/2020-10-03T22:14:06Z' failed to verify, so please scroll up in the log and find the parts containing the info about that snapshot. that the vm/208 group shows up in the log, but not in the contents likely just means that it has no snapshots at all.

 

[carsten2]

The defective backup snapshothas a ever running hour glass in the contents view

 

[carsten2]

the log says that 'ct/103/2020-10-03T22:14:06Z' failed to verify, so please scroll up in the log and find the parts containing the info about that snapshot. that the vm/208 group shows up in the log, but not in the contents likely just means that it has no snapshots at all.

Here the log:
2020-11-16T09:38:28+01:00: percentage done: 3.92% (44 of 1123 snapshots)
2020-11-16T09:38:28+01:00: verify pvebackup:ct/103/2020-10-03T22:14:06Z - manifest load error: unable to load blob '"/data1/pvebackup/ct/103/2020-10-03T22:14:06Z/index.json.blob"' - No such file or directory (os error 2)

Just delete this snapshot to fix it?

How to remove vm208 data? Will it go away after garbage collection?

 

[fabian]

yes, you can delete the snapshot. empty groups currently are not removed (they only contain the previous owner) as they are harmless.