[SOLVED] Backup fails with unknown cause

[Glowsome]

Hi all, i have a (current) standalone node with ProxMox where in advance of getting it into production to test i copies some VMs from the current prod server.
However for some unknown reason backup fails at 100%, and theres no real errorcode i can investigate/solve.
Both VMs are stopped, so locking shouldnt be an issue.

Common stuff checked prior to posting :

1. Storage/mountpoint to write to ( /data/backups ) contains the definitions in GUI to store VZDump files
2. The storage has more then enough space available (~800GB)
3. The total backup job contains 2 machines , nowhere near the full space available.
4. Even tho the backup job errors i do see a backup file of the 1st VM being processed, but it never gets to 2nd VM...

proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-15
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

The log from a/the backup run :

()
INFO: starting new backup job: vzdump --all 1 --storage backups --quiet 1 --compress gzip --mode snapshot --mailnotification failure
INFO: Starting Backup of VM 201 (qemu)
INFO: Backup started at 2020-01-08 01:00:01
INFO: status = stopped
INFO: update VM 201: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: nginx
INFO: include disk 'virtio0' 'vms:201/vm-201-disk-0.qcow2' 10G
INFO: creating archive '/data/backups/dump/vzdump-qemu-201-2020_01_08-01_00_01.vma.gz'
INFO: starting kvm to execute backup task
ERROR: Backup of VM 201 failed - volume 'iso:iso/CentOS-7-x86_64-Minimal-1611.iso' does not exist
INFO: Failed at 2020-01-08 01:00:04
INFO: Starting Backup of VM 301 (qemu)
INFO: Backup started at 2020-01-08 01:00:04
INFO: status = stopped
INFO: update VM 301: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: IDVault
INFO: include disk 'scsi0' 'vms:301/vm-301-disk-0.qcow2' 40G
INFO: creating archive '/data/backups/dump/vzdump-qemu-301-2020_01_08-01_00_04.vma.gz'
INFO: starting kvm to execute backup task
INFO: started backup task 'abb97e64-f0ba-4aad-a6cd-ca0a67d7403c'
INFO: status: 0% (54067200/42949672960), sparse 0% (2019328), duration 3, read/write 18/17 MB/s
INFO: status: 1% (437190656/42949672960), sparse 0% (3223552), duration 35, read/write 11/11 MB/s
INFO: status: 2% (862912512/42949672960), sparse 0% (3690496), duration 62, read/write 15/15 MB/s
INFO: status: 3% (1319043072/42949672960), sparse 0% (21221376), duration 83, read/write 21/20 MB/s
INFO: status: 4% (1733361664/42949672960), sparse 0% (38158336), duration 104, read/write 19/18 MB/s
INFO: status: 5% (2147680256/42949672960), sparse 0% (39690240), duration 125, read/write 19/19 MB/s
INFO: status: 6% (2581004288/42949672960), sparse 0% (39772160), duration 151, read/write 16/16 MB/s
INFO: status: 8% (3540320256/42949672960), sparse 1% (725704704), duration 167, read/write 59/17 MB/s
.............. (truncated to go below max post size)
INFO: status: 100% (42949672960/42949672960), sparse 45% (19720916992), duration 1276, read/write 1610/1 MB/s
INFO: transferred 42949 MB in 1276 seconds (33 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 13.01GB
INFO: delete old backup '/data/backups/dump/vzdump-qemu-301-2020_01_06-01_00_05.vma.gz'
INFO: Finished Backup of VM 301 (00:21:21)
INFO: Backup finished at 2020-01-08 01:21:25
INFO: Backup job finished with errors

Any guidance in this issue is greatly appreciated.

 

[Glowsome]

I found out why ... and the answer was 2 parts :

• a VM had an iso referenced which was not available on the new box
• a vm also had a bridge (vmbr1) reference which was not available on the new box

IMHO when importing a VM to a new box this stuff should be detected by the import-process and if not available either erase the reference or notify the admin so it gets corrected ... this has costed me sooo much time to find out ...