VM backup error

[danfmc]

Hello everyone.
I'm a noob user of Proxmox. I use it to have Home Assistant VM and several related LXC. I'm in version 7.4-16. Again, I'm a noob, my background is health services, but I want to learn more about Proxmox.
My VMs and LXCs are working fine, without issues. Only thing is... backup. I can make backup of my 7 LXCs without problem, but with my one and only VM, I can't do backups.
Short story even shorter: I had scheduled backups of VM and LXCs once a week (yes, probably I'm stupid) made to an external drive. Some weeks ago, when proxmox tries to backup the VM, it all freezes. If I try to make the backup manually, it stalls at 15% with following error:

ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 101 failed - job failed with err -5 - Input/output error
INFO: Failed at 2023-11-18 19:26:46
INFO: Backup job finished with errors
TASK ERROR: job errors

I get this error everytime.
Help?

 

[cwt]

You mentioned an external drive as backup target. Did you check this for errors? Or is there any concurrent access to the drive? Bad sectors or similar failures on external drives can lead to this symptoms.

 

[danfmc]

You mentioned an external drive as backup target. Did you check this for errors? Or is there any concurrent access to the drive? Bad sectors or similar failures on external drives can lead to this symptoms.

I use the drive only for proxmox backups. Probably bad drive, I tried to check it for errors, but can't find how. With

e2fsck -pf /dev/sdb1

I got "target is busy".

 

[cwt]

Running filechecks wouldn’t be enough. The drive is busy because it’s mounted. You should unmount it first.

I would recommend to run hardware tests with smartmon.

 

[danfmc]

I don't know if it is because it's an usb mechanical drive, but I can't run those tests as drive lacks SMART capability.
EDIT:
Short Test:
root@casacc:~# smartctl -t short /dev/sdb1 smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.116-1-pve] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org Short Background Self Test has begun Use smartctl -X to abort test root@casacc:~# smartctl -l selftest /dev/sdb1 smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.116-1-pve] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background short Completed - 0 - [- - -]
Long Test:

=== START OF READ SMART DATA SECTION ===
SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                   -       0                 - [-   -    -]

 

[danfmc]

Hello again.
Still have the same issue, but now with a brand new external ssd, in xfs file system, accepting VZDump backup, with lots of free space (2TB) for a 32GB VM backup.
Help please!

 

[danfmc]

So... now I tried to move the VM storage to another drive, I got error:
transferred 4.8 GiB of 32.0 GiB (15.01%) qemu-img: error while reading at byte 5163187200: Input/output error TASK ERROR: storage migration failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f raw -O qcow2 /dev/pve/vm-101-disk-1 zeroinit:/mnt/pve/SSDB/images/101/vm-101-disk-0.qcow2' failed: exit code 1

I tried to backup, got the err 5 input/output.

I tried to clone the VM, got error:
qemu-img: error while reading at byte 5163187200: Input/output error Logical volume "vm-106-disk-0" successfully removed Logical volume "vm-106-disk-1" successfully removed TASK ERROR: clone failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f raw -O raw /dev/pve/vm-101-disk-1 zeroinit:/dev/SSDA/vm-106-disk-1' failed: exit code 1

I can do snapshots without errors, I think! I got this result:
completed saving the VM state in 33s, saved 4.53 GiB snapshotting 'drive-scsi0' (local-lvm:vm-101-disk-1) Logical volume "snap_vm-101-disk-1_HAOS29122023" created. snapshotting 'drive-efidisk0' (local-lvm:vm-101-disk-0) Logical volume "snap_vm-101-disk-0_HAOS29122023" created. TASK OK
Is normal that my snapshot raw disk has 17,7GB instead of the 4,53 saved?

The ssd where my VM is stored has the following S.M.A.R.T. results:
=== START OF INFORMATION SECTION === Model Family: Phison Driven SSDs Device Model: KINGSTON SA400S37480G Serial Number: 50026B7282E286B2 LU WWN Device Id: 5 0026b7 282e286b2 Firmware Version: SBFKQ1.3 User Capacity: 480,103,981,056 bytes [480 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device TRIM Command: Available Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Dec 29 14:17:21 2023 WET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (65535) seconds. Offline data collection capabilities: (0x11) SMART execute Offline immediate. No Auto Offline data collection support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. No Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 2) minutes. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x0032 100 100 000 Old_age Always - 104 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 7413 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35 148 Unknown_Attribute 0x0000 100 100 000 Old_age Offline - 5 149 Unknown_Attribute 0x0000 100 100 000 Old_age Offline - 17 167 Write_Protect_Mode 0x0000 100 100 000 Old_age Offline - 0 168 SATA_Phy_Error_Count 0x0012 100 100 000 Old_age Always - 0 169 Bad_Block_Rate 0x0000 100 100 000 Old_age Offline - 14 170 Bad_Blk_Ct_Erl/Lat 0x0000 100 100 010 Old_age Offline - 0/11 172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0 173 MaxAvgErase_Ct 0x0000 100 100 000 Old_age Offline - 430 (Average 407) 181 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0 182 Erase_Fail_Count 0x0000 100 100 000 Old_age Offline - 4 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 4 192 Unsafe_Shutdown_Count 0x0012 100 100 000 Old_age Always - 26 194 Temperature_Celsius 0x0022 040 050 000 Old_age Always - 40 (Min/Max 15/50) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 8 199 SATA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 851999 218 CRC_Error_Count 0x0032 100 100 000 Old_age Always - 5 231 SSD_Life_Left 0x0000 072 072 000 Old_age Offline - 72 233 Flash_Writes_GiB 0x0032 100 100 000 Old_age Always - 62523 241 Lifetime_Writes_GiB 0x0032 100 100 000 Old_age Always - 66988 242 Lifetime_Reads_GiB 0x0032 100 100 000 Old_age Always - 179232 244 Average_Erase_Count 0x0000 100 100 000 Old_age Offline - 407 245 Max_Erase_Count 0x0000 100 100 000 Old_age Offline - 430 246 Total_Erase_Count 0x0000 100 100 000 Old_age Offline - 1455704 SMART Error Log not supported SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] Selective Self-tests/Logging not supported

 

[danfmc]

Ok. I think I managed to solve the problem. My problem was with a Home Assistant VM. Just made a backup inside HA, exported it, created another HA VM, used the backup, and there it is, now I can make backups with proxmox!