Snapshot Mode and Resume Not Working When Backup Compression=None

[Lonnie]

In Proxmox 7, I attempted to do a "snapshot mode backup" with compression set to "none". During the backup, the Virtual Machine was completely inaccessible and none of its services worked.

Also, I attempted doing a "stop mode backup" with compression set to "none". During the backup, I hit the Resume button, and the virtual machine would not resume.

Note: The physical server has 8 SSDs in a RAID 10 and is far from being over-allocated.

Upon reviewing the documentation, I see no mention of special consideration that must be made for backups having compression set to "none".

Questions:

1. Is snapshot-mode not possible when compression is set to "none"?
2. Is Resume not possible when when doing a stop-mode backup with compression "none"?

 

[fabian]

it sounds like your problem lies elsewhere rather than with disabled compression could you post the output of pveversion -v, the guest config, the backup task log and the system log (journalctl --since ... --until ...) from around the time of the backup?

 

[Lonnie]

pveversion -v

proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-9
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-11
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-10
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-3
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-14
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

Guest Config





journalctl --since="2021-10-11 21:17:09" --until="2021-10-11 22:06:05" > log.txt : (see attachment) @fabian

 

[fabian]

the backup task log and a few minutes before the backup started would also be interesting. thanks!

 

[Lonnie]

The attached log.txt starts 17 minutes prior to backup. @fabian

 

[fabian]

the backup task log is still missing..

 

[Lonnie]

Are you talking about the log you see in the Web interface during backup? There were two of these.

The first is from the "snapshot mode backup", where I was able to successfully stop the backup.

The 2nd is from when I tried a "stop mode backup", where I had a failed attempt to resume during that backup, and then I also could not stop the backup (it was not progressing anymore, but didn't show stopped status -- vm locked) until rebooted. See attachments. @fabian

I thought this had something to do with compression = none, because I don't have these issues when I do backups with compression.

 

[fabian]

okay. the logs indicate that starting the backup job overloads your system - what kind of storage are you using for the VM disks? and as backup target?

 

[Lonnie]

okay. the logs indicate that starting the backup job overloads your system - what kind of storage are you using for the VM disks? and as backup target?

The VM disk are virtio, as shown in the screenshots above.

For the backup target, I'm using an Ubuntu 20.04 NFS share, running on a intel core duo, with an enterprise hard drive that has plenty of free space. These types of backups succeed when compression does not equal none. Why would simply writing 4% of the file without compression overload that system, while writing the entire backup with compression does not overload the system? Why would Promox push any faster than can be received? Also, even if, why should a backup's failure prevent "snapshot mode backup" from running the virtual machine during the entire process? or prevent Resume for stop-mode-backups?@fabian

 

[fabian]

likely because the write load with compression is much lower than without, but that can only be told for sure by monitoring the resource usage while attempting backups. you can also attempt setting a bwlimit for the backup job to reduce the load.

 

[Lonnie]

likely because the write load with compression is much lower than without, but that can only be told for sure by monitoring the resource usage while attempting backups. you can also attempt setting a bwlimit for the backup job to reduce the load.

I've never encountered anything like this before. Typically, any sender will not send any faster than what is acknowledge by the receiver; the sender simply sends as fast as can be received and no faster than that. The TCP protocol has built-in features that regulate this type of thing.

Also, no matter what's wrong with a backup storage, failure of that storage shouldn't crash a running virtual machine. Instead, Proxmox should log the backup-failure without the virtual machine ever stopping. Robustness first!

Hmm, the virtual machine host is running 8 SSDs in a RAID 10. So, it can certainly access data a lot faster than what can be sent over the Gigabit network. Typically, network transfers max out at about 112 MB/sec on our local network (which is way slower than the SSDs can output).

Using the Linux desktop all these years, I have encountered large transfers causing degraded performance before. Maybe Proxmox itself is outputting data from the running virtual machine faster than it can also keep that virtual machine running, but I would think that the 8SSDs in the RAID 10 could handle this IO load just fine.

Have you tested doing backups with no compression to various storage types? I suspect "compression=none" is not tried very much and if more people did try it, I wouldn't be the only one with this issue.

 

[fabian]

without proper monitoring data I can't tell you where it's going wrong. if it works with compression but doesn't without, the likely culprit is that the increased write load on the backup target storage or the back pressure from not being able to write fast enough causes the issue.