Corrupted qcow2 images

anders_eken

New Member
Mar 13, 2024
2
0
1
We use Proxmox Backup Server and Client v3.3.1 to backup our KVM guests. These use qcow2 for the disks with ext4 as file system.

To ensure that the backed up images are valid, the backed up disks are mounted via proxmox-backup-client map after uploading to the Proxmox Backup Server and checked with qemu-img check.

We have now noticed that some of the uploaded disk-images are corrupt:

Code:
qemu-img: Could not open '/dev/loop0': Could not read L1 table: Input/output error

We investigated further and found that the image is indeed corrupt. Even after various repair attempts, the data could only be partially restored. To get to the bottom of the matter and rule out other causes, we shut down the VM and backed up the disk directly. Verification of the original source image before backup was successful, but the verification of the backed up image failed.

Configuration of the disk:

XML:
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
      <source file='/path/to/disk.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>


We were also able to reproduce the behavior on a hypervisor with Debian 12 and Ubuntu 24.04.

In the course of further analysis, we discovered that images from freshly provisioned KVM guests were primarily affected. This means that the following behavior can now be reliably reproduced (we are able to provide a valid qcow2 image that can safely reproduce the error) :

1. Create a new KVM Guest
2. Backup the KVM Guest
3. qemu-check fails

Workaround:

1. Create a new KVM Guest
2. Write data in the KVM guest (e.g. with 'dd if=/dev/urandom of=/tmp/testfile bs=1M count=2048')
3. Backup the KVM Guest
4. qemu-check works

Is this behavior known? What is the cause of it? Thank you in advance!
 
Hi and thanks for the report.

However, I can not reproduce this issue on my local setup. Here is what I tested for completeness:
Code:
qemu-img create -f qcow2 test.img 20G
proxmox-backup-client backup test.img:./test.img
proxmox-backup-client map <snapshot> test.img
qemu-img check /dev/loop0

# Result:
No errors were found on the image.
Image end offset: 262144

So please provide more information:
1. Create a new KVM Guest
What tooling did you use and which version?
2. Backup the KVM Guest
What exact command did you use to backup the guest?
2. Write data in the KVM guest (e.g. with 'dd if=/dev/urandom of=/tmp/testfile bs=1M count=2048')
What guest OS was used and what is mounted on /tmp?

Configuration of the disk:

<disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' discard='unmap'/> <source file='/path/to/disk.qcow2'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk>
This does not look like related to Proxmox tooling config, so what was used? Is this libvirt or the like?

Also, if you are doing a backup of a running system, you must freeze the guest filesystems using the qemu-guest-agent first, then create a snapshot of the qcow2 image [0], unfreeze the filesystem.

[0] https://wiki.qemu.org/Documentation/CreateSnapshot

Edit: Also I forgot: Did you verify the backup snapshot successfully on PBS after the backup run?
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!