VZDUMP always failing for I/O error, even though the VM is turned off

[Raikoryu]

Hi all,

I have a Proxmox 5.2.1 installed on my HP N54L server, with a Debian VM installed on it (among others).

I would like to backup my VMs before attempting to upgrade Proxmox (first to 5.4, then to 6.x).

Each time I try to VZDUMP the Debian VM, it systematically blocks on 54% and this message:

INFO: starting new backup job: vzdump 101 --node pve --storage backup --compress lzo --mode snapshot --remove 0
INFO: Starting Backup of VM 101 (qemu)
INFO: status = stopped
INFO: update VM 101: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: Debian
INFO: include disk 'scsi0' 'local-lvm:vm-101-disk-1' 32G
INFO: include disk 'sata0' '/dev/sdd1' 786408511897
INFO: creating archive '/backup/dump/vzdump-qemu-101-2020_02_21-15_59_48.vma.lzo'
INFO: starting kvm to execute backup task
INFO: started backup task 'f25c63b4-e081-4503-8d44-fa59d7ca9bb4'
INFO: status: 0% (292683776/820791738368), sparse 0% (151068672), duration 3, read/write 97/47 MB/s
INFO: status: 1% (8360558592/820791738368), sparse 0% (4589408256), duration 102, read/write 81/36 MB/s
INFO: status: 2% (16492331008/820791738368), sparse 1% (12720787456), duration 158, read/write 145/0 MB/s
INFO: status: 3% (24692457472/820791738368), sparse 2% (20920508416), duration 216, read/write 141/0 MB/s
INFO: status: 4% (32933478400/820791738368), sparse 3% (29161111552), duration 274, read/write 142/0 MB/s
[...]
INFO: status: 54% (450469625856/820791738368), sparse 54% (445594955776), duration 3232, read/write 99/0 MB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 101 failed - job failed with err -5 - Input/output error
INFO: Backup job finished with errors
TASK ERROR: job errors

I tried my luck on Google to check what I should do, but as a beginner, I couldn't find any answer specific enough to my situation.

I also tried to power off the VM, the result is the same.
My /backup folder is mapped on a ext4 partition of 1.3 TB on the same drive than the VM itself (which is on a 750 GB partition).

Thank you for your help.

Bonus: I also tried to exclude a drive from VZDUMP on another VM by putting

--exclude-path "/dev/sdc"

In the vzdump.conf, but the command is always ignored when I launch a VZDUMP (the drive is included).

I tried several formatting options such as

exclude-path: "/dev/sdc", removing quotes, adding the "/" at the end...etc. it just won't work. If you can help with that too, I would much appreciate.

 

[t.lamprecht]

I mean backups ain't a bad idea for sure, but at least to 5.4 the update should be straight forward. Actually you really should try to stay at the latest stable available - waiting out major updates for a bit can be OK, but the point releases are rolling ones and include lots of bug and security fixes..

It seems to have issues after writing about 420 GB, well is there enough space left on your target storage:
pvesm status

The input output error can be just a "no space left error", or also a real error showing a corrupt device (target or source).

 

[Raikoryu]

root@pve:~# pvesm status
Name Type Status Total Used Available %
backup dir active 1371329968 77924 1301522508 0.01%
local dir active 59600812 34067164 22476396 57.16%
local-lvm lvmthin active 154791936 0 154791936 0.00%

I mean backups ain't a bad idea for sure, but at least to 5.4 the update should be straight forward.

But HOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOW?
Everybody tells me that, but nobody tells me how to do it...
All answers point to this useless tutorial: https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0
Why is it useless? Simply because it explains nowhere how to update to 5.4 before being able to run the checklist...

apt-get update never EVER works!

root@pve:~# apt-get update
Get:1 http://security.debian.org stretch/updates InRelease [94.3 kB]
Hit:2 http://ftp.debian.org/debian buster InRelease
Get:3 http://ftp.debian.org/debian buster-updates InRelease [49.3 kB]
Get:4 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Ign:5 http://ftp.fr.debian.org/debian stretch InRelease
Get:6 http://ftp.fr.debian.org/debian stretch-updates InRelease [91.0 kB]
Hit:7 http://ftp.fr.debian.org/debian stretch Release
Get:8 http://download.proxmox.com/debian/pve buster InRelease [3,051 B]
Ign:9 https://enterprise.proxmox.com/debian/pve stretch InRelease
Ign:10 https://enterprise.proxmox.com/debian/pve stretch Release
Ign:11 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Pa ckages
Ign:12 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Pack ages
Ign:13 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en
Ign:14 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en_US
Ign:11 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Pa ckages
Ign:12 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Pack ages
Ign:13 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en
Ign:14 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en_US
Ign:11 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Pa ckages
Ign:12 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Pack ages
Ign:13 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en
Ign:14 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en_US
Ign:11 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Pa ckages
Ign:12 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Pack ages
Ign:13 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en
Ign:14 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en_US
Ign:11 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Pa ckages
Ign:12 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Pack ages
Ign:13 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en
Ign:14 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en_US
Err:11 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise amd64 Pa ckages
401 Unauthorized
Ign:12 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise all Pack ages
Ign:13 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en
Ign:14 https://enterprise.proxmox.com/debian/pve stretch/pve-enterprise Translat ion-en_US
Ign:8 http://download.proxmox.com/debian/pve buster InRelease
Fetched 303 kB in 1s (197 kB/s)
Reading package lists... Done
W: The repository 'https://enterprise.proxmox.com/debian/pve stretch Release' do es not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potential ly dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration deta ils.
W: GPG error: http://download.proxmox.com/debian/pve buster InRelease: The follo wing signatures couldn't be verified because the public key is not available: NO _PUBKEY 7BF2812E8A6E88E0
W: The repository 'http://download.proxmox.com/debian/pve buster InRelease' is n ot signed.
N: Data from such a repository can't be authenticated and is therefore potential ly dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration deta ils.
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/stretch/pve-e nterprise/binary-amd64/Packages 401 Unauthorized
E: Some index files failed to download. They have been ignored, or old ones used instead.
root@pve:~#

I already tried to add legit repos and a public key, but the result is the same...

I'd appreciate your patience despite my tone of exasperation.

 

[Raikoryu]

Okay, here's what I managed to do:

I edited my sources in /etc/apt/sources.list to remove the junk entreprise repos that need a licence and put this:

deb http://ftp.fr.debian.org/debian stretch main contrib
deb http://ftp.fr.debian.org/debian stretch-updates main contrib

deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

# security updates
deb http://security.debian.org/debian-security stretch/updates main contrib

Then I ran:

apt-get install dist-upgrade

And now I'm in 5.4.13!!!

I'll try VZDUMP again.

 

[Raikoryu]

Well, as far as I can see, I still have the same problem... :/