Hi,
I encountered an issue on my Proxmox VE system where newly created QCOW2 disks on classic LVM can contain stale data from previously used LVM extents.
Note: I encountered this issue on my Proxmox VE system during VM disk creation. I used ChatGPT to help analyze the behavior and to structure the controlled reproduction steps below. All commands and results shown below were actually tested on my system.
Environment
===========
Proxmox VE: 9.2.20
libpve-storage-perl: 9.1.10
pve-qemu-kvm: 11.0.3-3
qemu-img: 11.0.3
Kernel: 7.0.14-16-pve
The affected storage is classic LVM on a shared FC multipath LUN.
Storage configuration:
lvm: fc-vg-snap-l1
vgname fc-vg-snap
content rootdir,images
saferemove 0
shared 1
snapshot-as-volume-chain 1
Initial problem
===============
I added a new 96 GiB SCSI disk to a Windows Server 2025 VM.
Although the virtual disk had just been created by Proxmox, Windows detected an existing MBR partition table.
Even more suspiciously, one of the partitions extended beyond the size of the newly created 96 GiB virtual disk.
Reading guest LBA 0 showed an old MBR including the standard boot strings:
Invalid partition table
Error loading operating system
Missing operating system
I then created additional new disks to reproduce the problem.
One newly created 132 GiB disk appeared correctly as RAW in Windows.
Another newly created 64 GiB disk appeared as MBR and again contained an old partition table.
This made the problem reproducible but initially appeared random depending on which LVM extents were allocated.
Proxmox creation path
=====================
With:
snapshot-as-volume-chain 1
the LVM plugin uses QCOW2.
The installed LVMPlugin.pm contains:
my $options = {
preallocation => PVE::Storage:
lugin:reallocation_cmd_opt($scfg, $fmt),
};
Plugin.pm contains:
if ($fmt eq 'qcow2') {
$prealloc = $prealloc // 'metadata';
Therefore newly created QCOW2 images on this LVM storage are effectively created using:
qemu-img create -f qcow2 -o preallocation=metadata ...
I also tried configuring:
pvesm set fc-vg-snap-l1 --preallocation off
but PVE rejects this:
update storage failed: unexpected property 'preallocation'
Checking LVMPlugin:
ptions() confirms that "preallocation" is currently not exposed as an option for the LVM storage plugin.
Controlled reproduction without VM or Windows
=============================================
To exclude Windows, VirtIO SCSI, VM hotplug and the VM itself, I reproduced the behavior directly on a disposable LVM LV.
I created an 80 MiB test LV.
Before creating the QCOW2 image, I wrote a known marker directly into the underlying LV at offset 327680 (0x50000):
printf 'PROXMOX-QCOW2-STALE-DATA-TEST' | \
dd of=/dev/fc-vg-snap/debug-qcow2-pattern \
bs=1 seek=327680 conv=notrunc status=none
I verified that the marker existed at that raw LV offset.
Then I created a 64 MiB QCOW2 image on the same block device:
qemu-img create -f qcow2 -o preallocation=metadata \
/dev/fc-vg-snap/debug-qcow2-pattern 64M
qemu-img map showed that guest offset 0 was mapped to LV offset 327680.
Reading guest LBA 0 through QCOW2:
qemu-img dd \
if=/dev/fc-vg-snap/debug-qcow2-pattern \
of=/tmp/debug-pattern-sector0.bin \
bs=512 count=1
returned the marker:
PROXMOX-QCOW2-STALE-DATA-TEST
So data that existed in the LV BEFORE "qemu-img create" became visible as guest data in the newly created QCOW2 image.
Counter-test with preallocation=off
===================================
I then removed the test LV, recreated it, and wrote the same marker to the same raw offset.
This time I created the QCOW2 image with:
qemu-img create -f qcow2 -o preallocation=off \
/dev/fc-vg-snap/debug-qcow2-pattern 64M
Reading guest LBA 0 through QCOW2 now returned only zeros.
qemu-img map reported:
[{
"start": 0,
"length": 67108864,
"depth": 0,
"present": false,
"zero": true,
"data": false,
"compressed": false
}]
So the controlled A/B test produced:
preallocation=metadata
-> pre-existing data in the LV became visible as guest data
preallocation=off
-> unallocated guest sectors read as zero
FC / multipath checks
=====================
I also investigated whether the issue could be caused by inconsistent FC paths.
Direct-I/O reads of the relevant physical extents through both FC paths produced identical hashes.
The physical LVM extent mapping was also calculated and verified directly against the multipath-backed device.
I therefore found no evidence of an FC path inconsistency.
More importantly, the controlled marker test reproduces the problem directly on a single LVM LV without involving Windows or a running VM.
Possible data-remanence / security impact
=========================================
This seems potentially important from a data-remanence perspective.
Classic LVM can allocate extents that still contain data from a previously deleted LV.
If a new QCOW2 image is then created on that LV using:
preallocation=metadata
our controlled test shows that pre-existing bytes can become guest-visible data.
This explains why a newly created VM disk can suddenly contain an old MBR/partition table.
"saferemove 1" may help for LVs deleted after enabling it, but it does not sanitize extents that are already free.
In the controlled test, using:
preallocation=off
prevented the pre-existing underlying data from becoming visible to the guest.
Questions
=========
1. Is preallocation=metadata expected/supported for QCOW2 images directly on classic LVM block devices when using snapshot-as-volume-chain?
2. Is it expected QEMU behavior that pre-existing block-device contents become allocated/visible QCOW2 guest data with preallocation=metadata?
3. Should the Proxmox LVM plugin use preallocation=off for QCOW2 images in this configuration?
4. Alternatively, should the newly allocated LV/data area be zeroed or discarded before creating the QCOW2 image?
5. Is there an existing patch or recommended workaround for current Proxmox VE 9.2 installations?
I can provide the complete command/output sequence and further test results if required.
Thanks!
I encountered an issue on my Proxmox VE system where newly created QCOW2 disks on classic LVM can contain stale data from previously used LVM extents.
Note: I encountered this issue on my Proxmox VE system during VM disk creation. I used ChatGPT to help analyze the behavior and to structure the controlled reproduction steps below. All commands and results shown below were actually tested on my system.
Environment
===========
Proxmox VE: 9.2.20
libpve-storage-perl: 9.1.10
pve-qemu-kvm: 11.0.3-3
qemu-img: 11.0.3
Kernel: 7.0.14-16-pve
The affected storage is classic LVM on a shared FC multipath LUN.
Storage configuration:
lvm: fc-vg-snap-l1
vgname fc-vg-snap
content rootdir,images
saferemove 0
shared 1
snapshot-as-volume-chain 1
Initial problem
===============
I added a new 96 GiB SCSI disk to a Windows Server 2025 VM.
Although the virtual disk had just been created by Proxmox, Windows detected an existing MBR partition table.
Even more suspiciously, one of the partitions extended beyond the size of the newly created 96 GiB virtual disk.
Reading guest LBA 0 showed an old MBR including the standard boot strings:
Invalid partition table
Error loading operating system
Missing operating system
I then created additional new disks to reproduce the problem.
One newly created 132 GiB disk appeared correctly as RAW in Windows.
Another newly created 64 GiB disk appeared as MBR and again contained an old partition table.
This made the problem reproducible but initially appeared random depending on which LVM extents were allocated.
Proxmox creation path
=====================
With:
snapshot-as-volume-chain 1
the LVM plugin uses QCOW2.
The installed LVMPlugin.pm contains:
my $options = {
preallocation => PVE::Storage:
lugin:reallocation_cmd_opt($scfg, $fmt),};
Plugin.pm contains:
if ($fmt eq 'qcow2') {
$prealloc = $prealloc // 'metadata';
Therefore newly created QCOW2 images on this LVM storage are effectively created using:
qemu-img create -f qcow2 -o preallocation=metadata ...
I also tried configuring:
pvesm set fc-vg-snap-l1 --preallocation off
but PVE rejects this:
update storage failed: unexpected property 'preallocation'
Checking LVMPlugin:
ptions() confirms that "preallocation" is currently not exposed as an option for the LVM storage plugin.Controlled reproduction without VM or Windows
=============================================
To exclude Windows, VirtIO SCSI, VM hotplug and the VM itself, I reproduced the behavior directly on a disposable LVM LV.
I created an 80 MiB test LV.
Before creating the QCOW2 image, I wrote a known marker directly into the underlying LV at offset 327680 (0x50000):
printf 'PROXMOX-QCOW2-STALE-DATA-TEST' | \
dd of=/dev/fc-vg-snap/debug-qcow2-pattern \
bs=1 seek=327680 conv=notrunc status=none
I verified that the marker existed at that raw LV offset.
Then I created a 64 MiB QCOW2 image on the same block device:
qemu-img create -f qcow2 -o preallocation=metadata \
/dev/fc-vg-snap/debug-qcow2-pattern 64M
qemu-img map showed that guest offset 0 was mapped to LV offset 327680.
Reading guest LBA 0 through QCOW2:
qemu-img dd \
if=/dev/fc-vg-snap/debug-qcow2-pattern \
of=/tmp/debug-pattern-sector0.bin \
bs=512 count=1
returned the marker:
PROXMOX-QCOW2-STALE-DATA-TEST
So data that existed in the LV BEFORE "qemu-img create" became visible as guest data in the newly created QCOW2 image.
Counter-test with preallocation=off
===================================
I then removed the test LV, recreated it, and wrote the same marker to the same raw offset.
This time I created the QCOW2 image with:
qemu-img create -f qcow2 -o preallocation=off \
/dev/fc-vg-snap/debug-qcow2-pattern 64M
Reading guest LBA 0 through QCOW2 now returned only zeros.
qemu-img map reported:
[{
"start": 0,
"length": 67108864,
"depth": 0,
"present": false,
"zero": true,
"data": false,
"compressed": false
}]
So the controlled A/B test produced:
preallocation=metadata
-> pre-existing data in the LV became visible as guest data
preallocation=off
-> unallocated guest sectors read as zero
FC / multipath checks
=====================
I also investigated whether the issue could be caused by inconsistent FC paths.
Direct-I/O reads of the relevant physical extents through both FC paths produced identical hashes.
The physical LVM extent mapping was also calculated and verified directly against the multipath-backed device.
I therefore found no evidence of an FC path inconsistency.
More importantly, the controlled marker test reproduces the problem directly on a single LVM LV without involving Windows or a running VM.
Possible data-remanence / security impact
=========================================
This seems potentially important from a data-remanence perspective.
Classic LVM can allocate extents that still contain data from a previously deleted LV.
If a new QCOW2 image is then created on that LV using:
preallocation=metadata
our controlled test shows that pre-existing bytes can become guest-visible data.
This explains why a newly created VM disk can suddenly contain an old MBR/partition table.
"saferemove 1" may help for LVs deleted after enabling it, but it does not sanitize extents that are already free.
In the controlled test, using:
preallocation=off
prevented the pre-existing underlying data from becoming visible to the guest.
Questions
=========
1. Is preallocation=metadata expected/supported for QCOW2 images directly on classic LVM block devices when using snapshot-as-volume-chain?
2. Is it expected QEMU behavior that pre-existing block-device contents become allocated/visible QCOW2 guest data with preallocation=metadata?
3. Should the Proxmox LVM plugin use preallocation=off for QCOW2 images in this configuration?
4. Alternatively, should the newly allocated LV/data area be zeroed or discarded before creating the QCOW2 image?
5. Is there an existing patch or recommended workaround for current Proxmox VE 9.2 installations?
I can provide the complete command/output sequence and further test results if required.
Thanks!
