Backup VM error "job failed with err -61"

[dr_mat]

Hello. Help me plz. Can't create a backup VM102. Located in local-lvm (nvme0n1p3).

INFO: starting new backup job: vzdump 102 --mode suspend --compress zstd --remove 0 --node ProxmoxVE --storage hdd INFO: Starting Backup of VM 102 (qemu) INFO: Backup started at 2021-10-18 19:32:53 INFO: status = stopped INFO: backup mode: stop INFO: ionice priority: 7 INFO: VM Name: Test INFO: include disk 'sata0' 'local-lvm:vm-102-disk-0' 44G INFO: creating vzdump archive '/mnt/hdd/dump/vzdump-qemu-102-2021_10_18-19_32_53.vma.zst' INFO: starting kvm to execute backup task INFO: started backup task '874f820f-6be9-439a-a6f3-18c4799fc609' INFO: 0% (26.0 MiB of 44.0 GiB) in 1s, read: 26.0 MiB/s, write: 17.5 MiB/s ERROR: job failed with err -61 - No data available INFO: aborting backup job INFO: stopping kvm after backup task ERROR: Backup of VM 102 failed - job failed with err -61 - No data available INFO: Failed at 2021-10-18 19:32:56 INFO: Backup job finished with errors TASK ERROR: job errors

 

[dr_mat]

Сloning VM

create full clone of drive sata0 (local-lvm:vm-102-disk-0) Formatting '/mnt/hdd/images/103/vm-103-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=47244640256 lazy_refcounts=off refcount_bits=16 transferred 0.0 B of 44.0 GiB (0.00%) qemu-img: error while reading at byte 27262976: Input/output error TASK ERROR: clone failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f raw -O qcow2 /dev/pve/vm-102-disk-0 zeroinit:/mnt/hdd/images/103/vm-103-disk-0.qcow2' failed: exit code 1

 

[dr_mat]

any info

root@ProxmoxVE:~# pveversion -v proxmox-ve: 6.4-1 (running kernel: 5.4.140-1-pve) pve-manager: 6.4-13 (running version: 6.4-13/9f411e79) pve-kernel-5.4: 6.4-6 pve-kernel-helper: 6.4-6 pve-kernel-5.3: 6.1-6 pve-kernel-5.4.140-1-pve: 5.4.140-1 pve-kernel-5.4.128-1-pve: 5.4.128-2 pve-kernel-5.4.124-1-pve: 5.4.124-2 pve-kernel-5.4.106-1-pve: 5.4.106-1 pve-kernel-5.4.101-1-pve: 5.4.101-1 pve-kernel-5.3.18-3-pve: 5.3.18-3 pve-kernel-5.3.10-1-pve: 5.3.10-1 ceph-fuse: 12.2.11+dfsg1-2.1+b1 corosync: 3.1.2-pve1 criu: 3.11-3 glusterfs-client: 5.5-3 ifupdown: 0.8.35+pve1 ksm-control-daemon: 1.3-1 libjs-extjs: 6.0.1-10 libknet1: 1.20-pve1 libproxmox-acme-perl: 1.1.0 libproxmox-backup-qemu0: 1.1.0-1 libpve-access-control: 6.4-3 libpve-apiclient-perl: 3.1-3 libpve-common-perl: 6.4-3 libpve-guest-common-perl: 3.1-5 libpve-http-server-perl: 3.2-3 libpve-storage-perl: 6.4-1 libqb0: 1.0.5-1 libspice-server1: 0.14.2-4~pve6+1 lvm2: 2.03.02-pve4 lxc-pve: 4.0.6-2 lxcfs: 4.0.6-pve1 novnc-pve: 1.1.0-1 proxmox-backup-client: 1.1.13-2 proxmox-mini-journalreader: 1.1-1 proxmox-widget-toolkit: 2.6-1 pve-cluster: 6.4-1 pve-container: 3.3-6 pve-docs: 6.4-2 pve-edk2-firmware: 2.20200531-1 pve-firewall: 4.1-4 pve-firmware: 3.3-1 pve-ha-manager: 3.1-1 pve-i18n: 2.3-1 pve-qemu-kvm: 5.2.0-6 pve-xtermjs: 4.7.0-3 qemu-server: 6.4-2 smartmontools: 7.2-pve2 spiceterm: 3.1-1 vncterm: 1.6-2 zfsutils-linux: 2.0.5-pve1~bpo10+1 root@ProxmoxVE:~# qm showcmd 102 --pretty /usr/bin/kvm \ -id 102 \ -name Test \ -no-shutdown \ -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' \ -mon 'chardev=qmp,mode=control' \ -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \ -mon 'chardev=qmp-event,mode=control' \ -pidfile /var/run/qemu-server/102.pid \ -daemonize \ -smbios 'type=1,uuid=247feca9-3df4-4a0c-b7b1-d78b02dfc923' \ -smp '4,sockets=2,cores=2,maxcpus=4' \ -nodefaults \ -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \ -vnc unix:/var/run/qemu-server/102.vnc,password \ -no-hpet \ -cpu 'kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep' \ -m 8000 \ -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \ -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' \ -device 'vmgenid,guid=8fd020fb-7d40-4010-a801-2ef8f57bd4fd' \ -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \ -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' \ -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \ -device 'usb-host,bus=xhci.0,hostbus=3,hostport=1.1,id=usb0' \ -device 'VGA,id=vga,bus=pci.0,addr=0x2,edid=off' \ -iscsi 'initiator-name=iqn.1993-08.org.debian:01:409cd7ee9774' \ -drive 'file=/mnt/hdd/template/iso/debian-live-11.1.0-amd64-gnome_nonfree.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' \ -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101' \ -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' \ -drive 'file=/dev/pve/vm-102-disk-0,if=none,id=drive-sata0,format=raw,cache=none,aio=native,detect-zeroes=on' \ -device 'ide-hd,bus=ahci0.0,drive=drive-sata0,id=sata0,rotation_rate=1,bootindex=100' \ -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' \ -device 'e1000,mac=E2:1D:76:B2:BA:F3,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102' \ -rtc 'driftfix=slew,base=localtime' \ -machine 'type=pc-i440fx-5.1+pve0' \ -global 'kvm-pit.lost_tick_policy=discard'

 

[dr_mat]

How to make check disk (local-lvm:vm-102-disk-0)

 

[fiona]

Hi,
can you see any warnings/errors in /var/log/syslog? Please check that your physical disk is still healthy, e.g. smartctl -aH /dev/<your disk>. The virtual disk is in raw format, so it cannot really be checked (at least not with qemu-img).

EDIT: fix command

 

[dr_mat]

root @ ProxmoxVE: ~ # smartclt -aH /dev/nvme0n1 -bash: smartclt: команда не найдена корень @ ProxmoxVE: ~ # sudo smartclt -aH/dev/nvme0n1 -bash: sudo: команда не найдена корень @ ProxmoxVE: ~ # sudo smartclt -aH/dev/nvme0n1p3 -bash: sudo: команда не найдена [/ ICODE]

 

[fiona]

Sorry, it's smartctl.

 

[dr_mat]

=== START OF INFORMATION SECTION ===
Model Number: ADATA SX6000LNP
Serial Number: 2J5020127932
Firmware Version: V9002s16
PCI Vendor/Subsystem ID: 0x10ec
IEEE OUI Identifier: 0x00e04c
Controller ID: 1
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 128,035,676,160 [128 GB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Tue Oct 19 14:08:21 2021 +03
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0007): Security Format Frmw_DL
Optional NVM Commands (0x0014): DS_Mngmt Sav/Sel_Feat
Log Page Attributes (0x03): S/H_per_NS Cmd_Eff_Lg
Maximum Data Transfer Size: 32 Pages
Warning Comp. Temp. Threshold: 118 Celsius
Critical Comp. Temp. Threshold: 150 Celsius

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 8.00W - - 0 0 0 0 0 0
1 + 4.00W - - 1 1 1 1 0 0
2 + 3.00W - - 2 2 2 2 0 0
3 - 0.0128W - - 3 3 3 3 4000 8000
4 - 0.0080W - - 4 4 4 4 8000 30000

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 43 Celsius
Available Spare: 100%
Available Spare Threshold: 32%
Percentage Used: 5%
Data Units Read: 14,410,795 [7.37 TB]
Data Units Written: 5,709,233 [2.92 TB]
Host Read Commands: 120,566,306
Host Write Commands: 121,723,713
Controller Busy Time: 0
Power Cycles: 132
Power On Hours: 1,647
Unsafe Shutdowns: 89
Media and Data Integrity Errors: 7
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0

Error Information (NVMe Log 0x01, 8 of 8 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS
0 1 0 0x0000 0x0000 0x000 0 0 -
- [5 unused entries]
6 1219368206019409473 0 0x0000 0x0000 0x000 0 0 -

 

[fiona]

Media and Data Integrity Errors: 7

There are a few errors.

Maybe you can try using ddrescue for the image /dev/pve/vm-102-disk-0. It's available via the package gddrescue in Debian/Proxmox.