Backup in stop mode

[GerhardK]

Hello all,

I am relatively new to proxmox and I would like to configure backups.
My linux vms are backed up with the snapshot mode, but I got a windows vm which I would like to backup in stop mode.
Actually what I expected was:
1. Shutdown the VM
2. copy the disks to the defined backup space
3. Start the machine

But this is not how it actually works since this is what I found in the logs:

INFO: starting new backup job: vzdump 150 --mailto email@mail.com --storage backup --mailnotification always --mode stop --compress zstd --quiet 1
INFO: Starting Backup of VM 150 (qemu)
INFO: Backup started at 2020-07-12 03:00:02
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: WindowsVM
INFO: include disk 'scsi0' 'home:150/vm-150-disk-0.qcow2' 400G
INFO: stopping vm
INFO: skip unused drive 'home:150/vm-150-disk-2.qcow2' (not included into backup)
INFO: creating archive '/backup//dump/vzdump-qemu-150-2020_07_12-03_00_02.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '5d70a1f7-4525-461b-8db1-44f34ef5721b'
INFO: resuming VM again after 24 seconds
INFO: status: 0% (559677440/429496729600), sparse 0% (139644928), duration 3, read/write 186/140 MB/s
INFO: status: 1% (4358668288/429496729600), sparse 0% (250011648), duration 51, read/write 79/76 MB/s
INFO: status: 2% (8597733376/429496729600), sparse 0% (338714624), duration 103, read/write 81/79 MB/s
INFO: status: 3% (12998606848/429496729600), sparse 0% (481968128), duration 151, read/write 91/88 MB/s

...

INFO: status: 98% (424743469056/429496729600), sparse 9% (41245569024), duration 6356, read/write 9664/0 MB/s
zstd: /*stdout*\: Input/output error
INFO: status: 100% (429496729600/429496729600), sparse 10% (45998825472), duration 6357, read/write 4753/0 MB/s
INFO: transferred 429496 MB in 6357 seconds (67 MB/s)
ERROR: Backup of VM 150 failed - zstd --rsyncable --threads=1 failed - wrong exit status 1
INFO: Failed at 2020-07-12 04:47:03
INFO: Backup job finished with errors

TASK ERROR: job errors

So before the actual copy of the disks the VM is being resumed. As far as I understand this behaviour leads to the Error found at the bottom of the log:

 zstd: /*stdout*\: Input/output error

I think the early resume leads to a change in the filesize (or modification of the file) during the copy, so that the above mentioned error occurs.

I have also checked the proxmox wiki: https://pve.proxmox.com/wiki/Backup_and_Restore#_backup_modes
which states:

stop mode
This mode provides the highest consistency of the backup, at the cost of a short downtime in the VM operation. It works by executing an orderly shutdown of the VM, and then runs a background Qemu process to backup the VM data. After the backup is started, the VM goes to full operation mode if it was previously running. Consistency is guaranteed by using the live backup feature.

So i assume this is an expected behavior - unfortunately this leads to a modification of the disk during the copy and therefore to an error (i think). Is there any way, I can make a proper and consistent backup of this VM?

If it helps these are the package versions I am running:

proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve) pve-manager: 6.2-9 (running version: 6.2-9/4d363c5b) pve-kernel-5.4: 6.2-4 pve-kernel-helper: 6.2-4 pve-kernel-5.4.44-2-pve: 5.4.44-2 ceph-fuse: 12.2.11+dfsg1-2.1+b1 corosync: 3.0.4-pve1 criu: 3.11-3 glusterfs-client: 5.5-3 ifupdown: not correctly installed ifupdown2: 3.0.0-1+pve2 libjs-extjs: 6.0.1-10 libknet1: 1.16-pve1 libproxmox-acme-perl: 1.0.4 libpve-access-control: 6.1-2 libpve-apiclient-perl: 3.0-3 libpve-common-perl: 6.1-5 libpve-guest-common-perl: 3.0-11 libpve-http-server-perl: 3.0-6 libpve-storage-perl: 6.2-3 libqb0: 1.0.5-1 libspice-server1: 0.14.2-4~pve6+1 lvm2: 2.03.02-pve4 lxc-pve: 4.0.2-1 lxcfs: 4.0.3-pve3 novnc-pve: 1.1.0-1 proxmox-mini-journalreader: 1.1-1 proxmox-widget-toolkit: 2.2-9 pve-cluster: 6.1-8 pve-container: 3.1-10 pve-docs: 6.2-4 pve-edk2-firmware: 2.20200531-1 pve-firewall: 4.1-2 pve-firmware: 3.1-1 pve-ha-manager: 3.0-9 pve-i18n: 2.1-3 pve-qemu-kvm: 5.0.0-9 pve-xtermjs: 4.3.0-1 qemu-server: 6.2-8 smartmontools: 7.1-pve2 spiceterm: 3.1-1 vncterm: 1.6-1

Any kind of help is highly appreciated!

Thank you in advance!

 

[t.lamprecht]

Hi,

1. Shutdown the VM
2. copy the disks to the defined backup space
3. Start the machine

Stop mode backups work a bit differently: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_backup_modes

So i assume this is an expected behavior - unfortunately this leads to a modification of the disk during the copy and therefore to an error (i think).

No, the guest process is started with the guest CPU initially frozen, we then setup tracking of storage changes and only then let the guest run, if it then writes to the storage to a part not yet backed up it is intercepted and backed up first.

There error could mean that your target storage is full, had IO errors or possibly network issues, did you checked there.
(note only took a quick "sunday evening look", so I might have missed something else obvious)

 

[GerhardK]

Thank you for your quick reply t.lamprecht!
I doubt that the storage is full. I am using a Hetzner Storagebox for my backups (2TB) and the other Backup jobs also do not report any errors.
In fact the storage still got 1.3 TB of free space.
I am not in a hurry so just relax