backup fails with file too large and VM unreachable

[rschwarz]

We run proxmox 6.3-3. A scheduled backup fails for a single VM with:

...
NFO: 59% (2.4 TiB of 4.1 TiB) in 5h 37m 20s, read: 124.1 MiB/s, write: 122.1 MiB/s
zstd: error 25 : Write error : File too large (cannot write compressed block)
ERROR: VM 102 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: VM 102 qmp command 'backup-cancel' failed - unable to connect to VM 102 qmp socket - timeout after 5983 retries
ERROR: Backup of VM 102 failed - VM 102 qmp command 'query-backup' failed - got timeout
INFO: Failed at 2021-02-11 14:48:43
INFO: Backup job finished with errors
TASK ERROR: job errors

The backup has ample space. Unfortunately not only the backup failed, but the VM was left unresponsive, i.e. the console could not be opened after the failed backup.

 

[fiona]

Hi,
what kind of backup storage are you using? Is there a maximum file size for the underlying file system? Did the other backups fail for the same reason, i.e. zstd: error 25 : Write error : File too large (cannot write compressed block)?

 

[rschwarz]

Type is directory.
getconf FILESIZEBITS /var/local/backup
64
Other backups (smaller ones) go fine.

 

[fiona]

Could you share the output of findmnt? How big is the largest backup that still worked?

 

[rschwarz]

I am curious what you are able to see from the output of findmnt. The largest backup that worked was about 89GB.
After an upgrade on friday, which involved a new zstd package I ran the backup again over the weekend. This time with a different error message which I unfortunately cannot share because I needed to reboot the host. After the failed backup I could neither contact the failed VM by any means nor was I able to stop or reboot it. The host complained about unability to obtain a lock. Only after rebooting the host system I regained access to the VM.

 

[rschwarz]

An additional observation: The guest VM is configured to run from a LVM partition, which I thought will not support snapshots. Nevertheless the backup can be configured for snapshot mode. How would this work? Indeed the VM is only stopped for a short time and is running again when the backup proceeds. If I change the backup mode to stop, the VM also is only stopped for a short time and is running again while backup is still active.

 

[fiona]

I am curious what you are able to see from the output of findmnt.

The problem

├─/var/local/backup                   /dev/mapper/Backup-backup ext3       rw,relatime

The ext3 filesystem does not support files that big.

The largest backup that worked was about 89GB.
After an upgrade on friday, which involved a new zstd package I ran the backup again over the weekend. This time with a different error message which I unfortunately cannot share because I needed to reboot the host. After the failed backup I could neither contact the failed VM by any means nor was I able to stop or reboot it. The host complained about unability to obtain a lock. Only after rebooting the host system I regained access to the VM.

There should still be a task log for the recently failed backup job (in the bottom panel in the GUI, double click on the task entry to see the log).

An additional observation: The guest VM is configured to run from a LVM partition, which I thought will not support snapshots. Nevertheless the backup can be configured for snapshot mode. How would this work? Indeed the VM is only stopped for a short time and is running again when the backup proceeds. If I change the backup mode to stop, the VM also is only stopped for a short time and is running again while backup is still active.

For VM backups, snapshot mode does not refer to storage-level snapshot, but to QEMU-level snapshot. This is possible, because QEMU controls the I/O-layer. The same is true for the stop mode. See here for more information.

 

[rschwarz]

Ok, so 2 TiB should be supported and I possibly get above that limit. Have you a suggestion how to remedy the situation?

 

[rschwarz]

Will be converting to ext4 be enough?

 

[fiona]

Will be converting to ext4 be enough?

Yes, that should work for the file size problem.

 

[rschwarz]

Oh, proxmox automatically recreated the dump and images folders

 

[fiona]

Oh, proxmox automatically recreated the dump and images folders

Hopefully not before it was mounted again. You can use pvesm set <storageID> --is_mountpoint 1 to make PVE aware that it's a mountpoint and then it will only create directories when it really is mounted.

 

[rschwarz]

I mounted and then checked, finding the directories are there. So I am set?

 

[fiona]

Ok, then it should be fine. You can still add the is_mountpoint option if you want. Then PVE will detect that the storage is not there if it's not mounted for whatever reason.

 

[rschwarz]

Backup works now. Thank you for help!