ERROR: job failed with err -61 - No data available

L

liptech

Member
Jan 14, 2021
30
6
13
44
Brasil
Hey guys!
I use Proxmox 8.13 and currently my virtual machines are showing the error in the title, luckily I keep regular backups.

Apparently the virtual HD is damaged when performing the backup, but the machine continues to operate as if it had no problem of any kind.

As expected, my solution is always to restore the backup, but this is happening quite frequently.

Would anyone have an idea what could cause this?
________________________________________________________________________________________________________________________________________________________
INFO: Starting Backup of VM 104 (qemu)
INFO: Backup started at 2024-01-15 20:56:04
INFO: status = running
INFO: VM Name: EVE-NG-5
INFO: include disk 'scsi0' 'VMS:104/vm-104-disk-0.qcow2' 200G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/Externo/dump/vzdump-qemu-104-2024_01_15-20_56_04.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'b4952d2e-ba45-4f30-bb38-37ec7464bd44'
INFO: resuming VM again
INFO: 0% (638.0 MiB of 200.0 GiB) in 3s, read: 212.7 MiB/s, write: 165.9 MiB/s
INFO: 1% (2.1 GiB of 200.0 GiB) in 10s, read: 213.9 MiB/s, write: 130.4 MiB/s
INFO: 3% (6.3 GiB of 200.0 GiB) in 18s, read: 538.8 MiB/s, write: 178.5 MiB/s
INFO: 4% (8.4 GiB of 200.0 GiB) in 21s, read: 703.2 MiB/s, write: 144.4 MiB/s
INFO: 4% (9.6 GiB of 200.0 GiB) in 34s, read: 95.9 MiB/s, write: 90.9 MiB/s
ERROR: job failed with err -61 - No data available
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 104 failed - job failed with err -61 - No data available
INFO: Failed at 2024-01-15 20:56:38

grateful for the attention!
LIPTECH
 
  • Like
Reactions: AD342262
Please provide the output of pveversion -v and the journal from the time of the backup: journalctl --since '2024-01-15 20:50:00' --until '2024-01-15 21:00:00' > journal.txt

I'm not sure I understand it correctly, the backup breaks the disk even though the guest continues to run?
And you have to restore the disk from backup at this point to get a working disk again?
 
liptech said:
Hey guys!
I use Proxmox 8.13 and currently my virtual machines are showing the error in the title, luckily I keep regular backups.

Apparently the virtual HD is damaged when performing the backup, but the machine continues to operate as if it had no problem of any kind.

As expected, my solution is always to restore the backup, but this is happening quite frequently.

Would anyone have an idea what could cause this?
________________________________________________________________________________________________________________________________________________________
INFO: Starting Backup of VM 104 (qemu)
INFO: Backup started at 2024-01-15 20:56:04
INFO: status = running
INFO: VM Name: EVE-NG-5
INFO: include disk 'scsi0' 'VMS:104/vm-104-disk-0.qcow2' 200G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/Externo/dump/vzdump-qemu-104-2024_01_15-20_56_04.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'b4952d2e-ba45-4f30-bb38-37ec7464bd44'
INFO: resuming VM again
INFO: 0% (638.0 MiB of 200.0 GiB) in 3s, read: 212.7 MiB/s, write: 165.9 MiB/s
INFO: 1% (2.1 GiB of 200.0 GiB) in 10s, read: 213.9 MiB/s, write: 130.4 MiB/s
INFO: 3% (6.3 GiB of 200.0 GiB) in 18s, read: 538.8 MiB/s, write: 178.5 MiB/s
INFO: 4% (8.4 GiB of 200.0 GiB) in 21s, read: 703.2 MiB/s, write: 144.4 MiB/s
INFO: 4% (9.6 GiB of 200.0 GiB) in 34s, read: 95.9 MiB/s, write: 90.9 MiB/s
ERROR: job failed with err -61 - No data available
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 104 failed - job failed with err -61 - No data available
INFO: Failed at 2024-01-15 20:56:38

grateful for the attention!
LIPTECH
Click to expand...
I have the same problem - new to proxmox - and first try to save a test vm
So very interested to the solution
 
mira said:
Please provide the output of pveversion -v and the journal from the time of the backup: journalctl --since '2024-01-15 20:50:00' --until '2024-01-15 21:00:00' > journal.txt

I'm not sure I understand it correctly, the backup breaks the disk even though the guest continues to run?
And you have to restore the disk from backup at this point to get a working disk again?
Click to expand...
Correct, the backup fails by accusing that the disc is corrupted, in the virtual machine I perform FSCK and VM detects no errors, but Proxmox cannot perform the backup.

Forcing me to restore the virtual machine.

Thanks in advance
Felipe Mendes
 

Attachments

  • journal.txt
    14.9 KB · Views: 3
  • pveversion.txt
    1.9 KB · Views: 3
Thank you for the files!

In the journal we can see an error:
Jan 15 20:56:36 fenix kernel: nvme0n1: I/O Cmd(0x2) @ LBA 193988344, 256 blocks, I/O Error (sct 0x2 / sc 0x81)
Jan 15 20:56:36 fenix kernel: critical medium error, dev nvme0n1, sector 193988344 op 0x0:(READ) flags 0x4000 phys_seg 11 prio class 2
Jan 15 20:56:37 fenix kernel: nvme0n1: I/O Cmd(0x2) @ LBA 193994232, 256 blocks, I/O Error (sct 0x2 / sc 0x81)
Jan 15 20:56:37 fenix kernel: critical medium error, dev nvme0n1, sector 193994232 op 0x0:(READ) flags 0x4000 phys_seg 3 prio class 2
Jan 15 20:56:38 fenix pvedaemon[3270959]: ERROR: Backup of VM 104 failed - job failed with err -61 - No data available
Click to expand...
It seems your NVMe is failing.
 
Hi, I have the same problem with a new install of last version of proxmox and I don't think my nvme is failing (tested on an other machine).
Could it be a bug ?
Best regards

root@proxmox:~# pveversion -v proxmox-ve: 8.1.0 (running kernel: 6.5.11-4-pve) pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15) proxmox-kernel-helper: 8.0.9 proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4 proxmox-kernel-6.5: 6.5.11-4 ceph-fuse: 17.2.7-pve1 corosync: 3.1.7-pve3 criu: 3.17.1-2 glusterfs-client: 10.3-5 ifupdown2: 3.2.0-1+pmx7 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-4 libknet1: 1.28-pve1 libproxmox-acme-perl: 1.5.0 libproxmox-backup-qemu0: 1.4.0 libproxmox-rs-perl: 0.3.1 libpve-access-control: 8.0.7 libpve-apiclient-perl: 3.3.1 libpve-common-perl: 8.1.0 libpve-guest-common-perl: 5.0.6 libpve-http-server-perl: 5.0.5 libpve-network-perl: 0.9.4 libpve-rs-perl: 0.8.7 libpve-storage-perl: 8.0.5 libspice-server1: 0.15.1-1 lvm2: 2.03.16-2 lxc-pve: 5.0.2-4 lxcfs: 5.0.3-pve3 novnc-pve: 1.4.0-3 proxmox-backup-client: 3.0.4-1 proxmox-backup-file-restore: 3.0.4-1 proxmox-kernel-helper: 8.0.9 proxmox-mail-forward: 0.2.2 proxmox-mini-journalreader: 1.4.0 proxmox-offline-mirror-helper: 0.6.2 proxmox-widget-toolkit: 4.1.3 pve-cluster: 8.0.5 pve-container: 5.0.8 pve-docs: 8.1.3 pve-edk2-firmware: 4.2023.08-1 pve-firewall: 5.0.3 pve-firmware: 3.9-1 pve-ha-manager: 4.0.3 pve-i18n: 3.1.2 pve-qemu-kvm: 8.1.2-4 pve-xtermjs: 5.3.0-2 qemu-server: 8.0.10 smartmontools: 7.3-pve1 spiceterm: 3.3.0 swtpm: 0.8.0+pve1 vncterm: 1.8.0 zfsutils-linux: 2.2.0-pve3
 
AD342262 said:
Hi, I have the same problem with a new install of last version of proxmox and I don't think my nvme is failing (tested on an other machine).
Could it be a bug ?
Best regards

root@proxmox:~# pveversion -v proxmox-ve: 8.1.0 (running kernel: 6.5.11-4-pve) pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15) proxmox-kernel-helper: 8.0.9 proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4 proxmox-kernel-6.5: 6.5.11-4 ceph-fuse: 17.2.7-pve1 corosync: 3.1.7-pve3 criu: 3.17.1-2 glusterfs-client: 10.3-5 ifupdown2: 3.2.0-1+pmx7 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-4 libknet1: 1.28-pve1 libproxmox-acme-perl: 1.5.0 libproxmox-backup-qemu0: 1.4.0 libproxmox-rs-perl: 0.3.1 libpve-access-control: 8.0.7 libpve-apiclient-perl: 3.3.1 libpve-common-perl: 8.1.0 libpve-guest-common-perl: 5.0.6 libpve-http-server-perl: 5.0.5 libpve-network-perl: 0.9.4 libpve-rs-perl: 0.8.7 libpve-storage-perl: 8.0.5 libspice-server1: 0.15.1-1 lvm2: 2.03.16-2 lxc-pve: 5.0.2-4 lxcfs: 5.0.3-pve3 novnc-pve: 1.4.0-3 proxmox-backup-client: 3.0.4-1 proxmox-backup-file-restore: 3.0.4-1 proxmox-kernel-helper: 8.0.9 proxmox-mail-forward: 0.2.2 proxmox-mini-journalreader: 1.4.0 proxmox-offline-mirror-helper: 0.6.2 proxmox-widget-toolkit: 4.1.3 pve-cluster: 8.0.5 pve-container: 5.0.8 pve-docs: 8.1.3 pve-edk2-firmware: 4.2023.08-1 pve-firewall: 5.0.3 pve-firmware: 3.9-1 pve-ha-manager: 4.0.3 pve-i18n: 3.1.2 pve-qemu-kvm: 8.1.2-4 pve-xtermjs: 5.3.0-2 qemu-server: 8.0.10 smartmontools: 7.3-pve1 spiceterm: 3.3.0 swtpm: 0.8.0+pve1 vncterm: 1.8.0 zfsutils-linux: 2.2.0-pve3
Click to expand...
Please provide the complete task log of a failing backup and the journal starting before the backup and ending afterwards (~5-10 minutes before and after).
 
please find them here


journalctl
Jan 17 15:17:02 proxmox kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Jan 17 15:17:02 proxmox kernel: fwpr100p0 (unregistering): left promiscuous mode
Jan 17 15:17:02 proxmox kernel: fwpr100p0 (unregistering): left allmulticast mode
Jan 17 15:17:02 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Jan 17 15:17:02 proxmox kernel: fwln100i0 (unregistering): left promiscuous mode
Jan 17 15:17:02 proxmox kernel: fwln100i0 (unregistering): left allmulticast mode
Jan 17 15:17:02 proxmox kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Jan 17 15:17:02 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Jan 17 15:17:02 proxmox kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Jan 17 15:17:02 proxmox kernel: tap100i0: left allmulticast mode
Jan 17 15:17:01 proxmox CRON[7053]: pam_unix(cron:session): session closed for user root
Jan 17 15:17:01 proxmox CRON[7054]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 15:17:01 proxmox CRON[7053]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jan 17 15:16:40 proxmox kernel: critical medium error, dev nvme0n1, sector 234637952 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 2
Jan 17 15:16:40 proxmox kernel: nvme0n1: I/O Cmd(0x2) @ LBA 234637952, 128 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR
Jan 17 15:16:30 proxmox pvedaemon[1025]: worker 6964 started
Jan 17 15:16:30 proxmox pvedaemon[1025]: starting 1 worker(s)
Jan 17 15:16:30 proxmox pvedaemon[1025]: worker 1026 finished
Jan 17 15:16:30 proxmox pvedaemon[1026]: worker exit
Jan 17 15:16:00 proxmox kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Jan 17 15:16:00 proxmox kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Jan 17 15:16:00 proxmox kernel: tap100i0: entered allmulticast mode
Jan 17 15:16:00 proxmox kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Jan 17 15:16:00 proxmox kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Jan 17 15:16:00 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Jan 17 15:16:00 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Jan 17 15:16:00 proxmox kernel: fwln100i0: entered promiscuous mode
Jan 17 15:16:00 proxmox kernel: fwln100i0: entered allmulticast mode
Jan 17 15:16:00 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Jan 17 15:16:00 proxmox kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Jan 17 15:16:00 proxmox kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Jan 17 15:16:00 proxmox kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Jan 17 15:16:00 proxmox kernel: fwpr100p0: entered promiscuous mode
Jan 17 15:16:00 proxmox kernel: fwpr100p0: entered allmulticast mode
Jan 17 15:16:00 proxmox kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Jan 17 15:16:00 proxmox kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Jan 17 15:15:59 proxmox kernel: tap100i0: entered promiscuous mode
Jan 17 15:15:59 proxmox systemd[1]: Started 100.scope.
Jan 17 15:15:58 proxmox pvedaemon[6773]: INFO: Starting Backup of VM 100 (qemu)
Jan 17 15:15:58 proxmox pvedaemon[6773]: INFO: starting new backup job: vzdump 100 --mode snapshot --storage DS1821PA2ZR --node proxmox --compress zstd --notificati>
Jan 17 15:15:58 proxmox pvedaemon[6621]: <root@pam> starting task UPID:proxmox:00001A75:00025A9E:65A7E11E:vzdump:100:root@pam:
 

Attachments

  • task-proxmox-vzdump-2024-01-17T14_15_58Z.log
    2 KB · Views: 7
Thanks for the journal and the task log!
Looks like it's the same issue, as we can see here:
Jan 17 15:16:40 proxmox kernel: critical medium error, dev nvme0n1, sector 234637952 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 2
Jan 17 15:16:40 proxmox kernel: nvme0n1: I/O Cmd(0x2) @ LBA 234637952, 128 blocks, I/O Error (sct 0x2 / sc 0x81) MORE DNR
Click to expand...
 
mira said:
Thanks for the journal and the task log!
Looks like it's the same issue, as we can see here:
Click to expand...
Hello, same issue. I saw this error first time today after updating from 8.0.4 to 8.1.4, the nvme drive has no errors, I think this has something to do with the new kernel or something similar.

Code: 
smartctl -a /dev/nvme0
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.5.11-8-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       SSD 512GB
Serial Number:                      202304132450
Firmware Version:                   VC2S038B
PCI Vendor/Subsystem ID:            0x10ec
IEEE OUI Identifier:                0x00e04c
Controller ID:                      1
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            00e04c 106f9f6d65
Local Time is:                      Fri Feb  9 22:30:47 2024 CET
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0054):     DS_Mngmt Sav/Sel_Feat Timestmp
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     100 Celsius
Critical Comp. Temp. Threshold:     110 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.00W       -        -    0  0  0  0        0       0
 1 +     4.00W       -        -    1  1  1  1        0       0
 2 +     3.00W       -        -    2  2  2  2        0       0
 3 -   0.0300W       -        -    3  3  3  3     5000   10000
 4 -   0.0050W       -        -    4  4  4  4    54000   45000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        40 Celsius
Available Spare:                    99%
Available Spare Threshold:          32%
Percentage Used:                    0%
Data Units Read:                    6,988,425 [3.57 TB]
Data Units Written:                 14,658,554 [7.50 TB]
Host Read Commands:                 101,537,939
Host Write Commands:                679,638,449
Controller Busy Time:               0
Power Cycles:                       20
Power On Hours:                     5,415
Unsafe Shutdowns:                   7
Media and Data Integrity Errors:    858
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 8 of 8 entries)
No Errors Logged

All the VMs I have run perfectly, but I can't back them up. The LXCs have some issues, can't create/can't restore.

The critical medium error spam kinda disappeared, returns only when I try to backup or run a LXC.
Weird coincidence we all get these nvme issues.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back