job failed with err -61 - No data available

[cphautomation]

Hi

I have an issue with my weekly backups made to a Synology NAS. They have started to fail with:

 job failed with err -61 - No data available

I'm running Virtual Environment 6.4-14 on a NUC with a Samsung SSD 970 EVO 500GB:

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 59 Celsius
Available Spare: 91%
Available Spare Threshold: 10%
Percentage Used: 1%
Data Units Read: 18,542,668 [9.49 TB]
Data Units Written: 34,279,454 [17.5 TB]
Host Read Commands: 231,027,780
Host Write Commands: 872,379,620
Controller Busy Time: 4,376
Power Cycles: 38
Power On Hours: 8,277
Unsafe Shutdowns: 26
Media and Data Integrity Errors: 612
Error Information Log Entries: 617
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 59 Celsius
Temperature Sensor 2: 60 Celsius

I also ran a bad block check the other day:

root@nuc:~# badblocks -v /dev/nvme0n1 > ~/bad_sectors.txt
Checking blocks 0 to 488386583
Checking for bad blocks (read-only test):
done                                                 
Pass completed, 522 bad blocks found. (522/0/0 errors)

From what I understand 522 is nothing to worry about on a SSD? What are my options here to get backup working again?

 

[shrdlicka]

Hi,

did all or only some of the backups fail with this error? Do you see errors in your syslog?

 

[cphautomation]

I run weekly backups. Last succesfull was May 10th. Since then it has failed twice. It has been failing on and off a couple of times this year as well.

This is the log:

Task viewer: VM/CT 100 - Backup

OutputStatus

Stop
INFO: starting new backup job: vzdump 100 --mailnotification failure --mode snapshot --mailto XXXXXXXXX --storage Synology --compress zstd --quiet 1 --node nuc
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-05-24 00:00:02
INFO: status = running
INFO: VM Name: hassosova-4.11
INFO: include disk 'sata0' 'local-lvm:vm-100-disk-1' 60G
INFO: include disk 'efidisk0' 'local-lvm:vm-100-disk-0' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/Synology/dump/vzdump-qemu-100-2022_05_24-00_00_02.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '71f807d5-c863-4da4-8632-e9f8ac5e8896'
INFO: resuming VM again
INFO: 0% (199.4 MiB of 60.0 GiB) in 3s, read: 66.5 MiB/s, write: 55.1 MiB/s
INFO: 1% (761.2 MiB of 60.0 GiB) in 10s, read: 80.3 MiB/s, write: 58.8 MiB/s
INFO: 2% (1.2 GiB of 60.0 GiB) in 16s, read: 85.2 MiB/s, write: 84.9 MiB/s
INFO: 3% (1.8 GiB of 60.0 GiB) in 24s, read: 73.9 MiB/s, write: 71.1 MiB/s
INFO: 4% (2.4 GiB of 60.0 GiB) in 32s, read: 77.9 MiB/s, write: 77.4 MiB/s
INFO: 5% (3.0 GiB of 60.0 GiB) in 40s, read: 76.1 MiB/s, write: 75.9 MiB/s
INFO: 6% (3.6 GiB of 60.0 GiB) in 47s, read: 88.0 MiB/s, write: 86.8 MiB/s
INFO: 7% (4.3 GiB of 60.0 GiB) in 55s, read: 80.7 MiB/s, write: 77.7 MiB/s
INFO: 8% (4.8 GiB of 60.0 GiB) in 1m 2s, read: 83.4 MiB/s, write: 83.1 MiB/s
INFO: 9% (5.5 GiB of 60.0 GiB) in 1m 10s, read: 81.6 MiB/s, write: 81.4 MiB/s
INFO: 10% (6.1 GiB of 60.0 GiB) in 1m 17s, read: 86.5 MiB/s, write: 86.4 MiB/s
INFO: 11% (6.6 GiB of 60.0 GiB) in 1m 23s, read: 93.6 MiB/s, write: 93.4 MiB/s
INFO: 12% (7.2 GiB of 60.0 GiB) in 1m 29s, read: 102.1 MiB/s, write: 101.7 MiB/s
INFO: 13% (7.9 GiB of 60.0 GiB) in 1m 37s, read: 87.9 MiB/s, write: 87.8 MiB/s
INFO: 14% (8.4 GiB of 60.0 GiB) in 1m 44s, read: 79.8 MiB/s, write: 78.4 MiB/s
INFO: 15% (9.0 GiB of 60.0 GiB) in 1m 52s, read: 73.4 MiB/s, write: 70.4 MiB/s
INFO: 16% (9.6 GiB of 60.0 GiB) in 1m 59s, read: 88.0 MiB/s, write: 87.9 MiB/s
INFO: 17% (10.2 GiB of 60.0 GiB) in 2m 8s, read: 70.9 MiB/s, write: 70.8 MiB/s
INFO: 18% (10.8 GiB of 60.0 GiB) in 2m 17s, read: 68.5 MiB/s, write: 65.2 MiB/s
INFO: 19% (11.4 GiB of 60.0 GiB) in 2m 24s, read: 84.4 MiB/s, write: 83.9 MiB/s
INFO: 20% (12.0 GiB of 60.0 GiB) in 2m 32s, read: 80.7 MiB/s, write: 80.6 MiB/s
INFO: 21% (12.6 GiB of 60.0 GiB) in 2m 41s, read: 65.7 MiB/s, write: 65.6 MiB/s
INFO: 22% (13.2 GiB of 60.0 GiB) in 2m 50s, read: 68.1 MiB/s, write: 65.4 MiB/s
INFO: 23% (13.8 GiB of 60.0 GiB) in 3m, read: 63.4 MiB/s, write: 63.3 MiB/s
INFO: 24% (14.5 GiB of 60.0 GiB) in 3m 10s, read: 64.5 MiB/s, write: 64.4 MiB/s
INFO: 25% (15.0 GiB of 60.0 GiB) in 3m 18s, read: 74.8 MiB/s, write: 71.6 MiB/s
INFO: 26% (15.6 GiB of 60.0 GiB) in 3m 27s, read: 64.0 MiB/s, write: 63.7 MiB/s
INFO: 27% (16.3 GiB of 60.0 GiB) in 3m 38s, read: 59.6 MiB/s, write: 59.0 MiB/s
INFO: 28% (16.9 GiB of 60.0 GiB) in 3m 48s, read: 61.3 MiB/s, write: 58.8 MiB/s
INFO: 29% (17.5 GiB of 60.0 GiB) in 3m 56s, read: 82.0 MiB/s, write: 82.0 MiB/s
INFO: 30% (18.0 GiB of 60.0 GiB) in 4m 4s, read: 70.2 MiB/s, write: 69.7 MiB/s
INFO: 31% (18.7 GiB of 60.0 GiB) in 4m 14s, read: 66.0 MiB/s, write: 65.9 MiB/s
INFO: 32% (19.3 GiB of 60.0 GiB) in 4m 20s, read: 100.9 MiB/s, write: 96.1 MiB/s
INFO: 33% (19.8 GiB of 60.0 GiB) in 4m 27s, read: 80.8 MiB/s, write: 80.8 MiB/s
INFO: 34% (20.5 GiB of 60.0 GiB) in 4m 35s, read: 81.1 MiB/s, write: 80.8 MiB/s
INFO: 35% (21.1 GiB of 60.0 GiB) in 4m 44s, read: 69.3 MiB/s, write: 66.1 MiB/s
INFO: 36% (21.6 GiB of 60.0 GiB) in 4m 53s, read: 64.0 MiB/s, write: 64.0 MiB/s
INFO: 37% (22.2 GiB of 60.0 GiB) in 5m 3s, read: 61.0 MiB/s, write: 60.0 MiB/s
INFO: 38% (22.8 GiB of 60.0 GiB) in 5m 13s, read: 58.9 MiB/s, write: 56.5 MiB/s
INFO: 38% (23.3 GiB of 60.0 GiB) in 5m 22s, read: 53.9 MiB/s, write: 53.3 MiB/s
ERROR: job failed with err -61 - No data available
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - job failed with err -61 - No data available
INFO: Failed at 2022-05-24 00:05:25
INFO: Backup job finished with errors

TASK ERROR: job errors

 

[shrdlicka]

From what I can gather it is most likely the SSD . There might be a slight possibility that it is an issue with writing to the NAS. You could try to backup to a different local disk and see if this fixes the issue. If it doesn't the SSD would be my main suspect.

 

[cphautomation]

From what I can gather it is most likely the SSD . There might be a slight possibility that it is an issue with writing to the NAS. You could try to backup to a different local disk and see if this fixes the issue. If it doesn't the SSD would be my main suspect.

I tried a backup directly to SSD with no issues. I then restarted NAS and attempted again. This time backup was also succesfull. From that I would presume a software issue with NAS. Either way thanks for the input as it pushed me in the right direction. The error message made me think it was a local issue.