data corruption on shared-lvm

grisu68 · Thursday at 12:01

Hello,

i have i weird situation with one of our shared-FC-storages

given:
3 nodes, each connected to a FC-switches ( 2 HBA-ports each, multipath)
2 storage-systems in the backend - configured a shared-storage on all 3 nodes
- ds1 - a HPE MSA2050
- san1 - a HUAWAI (provided by our hoster)

setup:
a VM (id 114) with 4 data-disks
/tst1 - on the local-store
/tst2 - on ds1 with qcow2
/tst3 - on ds1 with raw
/tst4 - on san1 with qcow2

issue:
file-corruption after copy from local store to a disk ds1
the file-size is the same in all tst-dirs
xfs-filesystem

Bash:

cp /tst1/COMPLETE_DATA_BACKUP_databackup_3_1 /tst2/
cp /tst1/COMPLETE_DATA_BACKUP_databackup_3_1 /tst3/
cp /tst1/COMPLETE_DATA_BACKUP_databackup_3_1 /tst4/

Bash:

vmsrv200:~ #  b3sum /tst1/COMPLETE_DATA_BACKUP_databackup_3_1
efef9d6ccee6910c66f7d3d7118ef6fc573bc4ded598d156b6448f3f671c5a42  /tst1/COMPLETE_DATA_BACKUP_databackup_3_1
vmsrv200:~ #  b3sum /tst2/COMPLETE_DATA_BACKUP_databackup_3_1
e173c44f91cc3a92cb154de0bba0e6282d49671669b23033d3951aada77c5602  /tst2/COMPLETE_DATA_BACKUP_databackup_3_1
vmsrv200:~ #  b3sum /tst3/COMPLETE_DATA_BACKUP_databackup_3_1
5daae12d1de773854c02224fc4868a24e8d26320299e44eea05414fcfd31df17  /tst3/COMPLETE_DATA_BACKUP_databackup_3_1
vmsrv200:~ #  b3sum /tst4/COMPLETE_DATA_BACKUP_databackup_3_1
efef9d6ccee6910c66f7d3d7118ef6fc573bc4ded598d156b6448f3f671c5a42  /tst4/COMPLETE_DATA_BACKUP_databackup_3_1

so /tst1 and /tst4 are OK - also the same checksum on the production-system
but on /tst2 and /tst3 the checksum is different

additional config-info:

Code:

root@pve2:/etc/pve# pveversion
pve-manager/9.2.10/43df2e01f27a1a19 (running kernel: 7.0.14-12-pve)

output a little bit stripped ...

Bash:

root@pve2:/etc/pve# cat storage.cfg

lvm: san1
    vgname san1
    content images,rootdir
    nodes pve3,pve1,pve2
    saferemove 0
    shared 1
    snapshot-as-volume-chain 1

lvm: ds1
    vgname vg_ds1
    content rootdir,images
    saferemove 0
    shared 1
    snapshot-as-volume-chain 1

lvmthin: pve2_local
    thinpool pve2_local
    vgname pve2_local
    content images,rootdir
    nodes pve2

Bash:

root@pve2:/etc/pve# qm config   114
agent: 1
bios: seabios
boot: order=scsi0
cores: 2
cpu: x86-64-v2-AES
hotplug: disk,network,usb,memory,cpu
memory: 12288
meta: creation-qemu=10.0.2,ctime=1782455951
name: vmsrv200
net0: virtio=00:50:56:85:21:05,bridge=prod2
numa: 1
ostype: l26
scsi0: ds1:vm-114-disk-0.qcow2,discard=on,iothread=1,size=150G,ssd=1
scsi1: pve2_local:vm-114-disk-0,backup=0,discard=on,iothread=1,size=100G,ssd=1
scsi2: ds1:vm-114-disk-1.qcow2,backup=0,discard=on,iothread=1,size=100G,ssd=1
scsi3: ds1:vm-114-disk-2,backup=0,discard=on,iothread=1,size=100G,ssd=1
scsi4: san1:vm-114-disk-0.qcow2,size=100G
scsihw: virtio-scsi-pci
smbios1: uuid=42051773-39af-d350-9ee1-e50c4bc5153b
sockets: 2
tags: 
vmgenid: a3585bbf-9d4c-438d-9caf-7e6c6a117d04

Bash:

root@pve2:/etc/pve# pvesm list ds1
Volid                   Format  Type              Size VMID
ds1:vm-114-disk-0.qcow2 qcow2   images    161061273600 114
ds1:vm-114-disk-1.qcow2 qcow2   images    107374182400 114
ds1:vm-114-disk-2       raw     images    107374182400 114

Bash:

root@pve2:/etc/pve# multipath -ll
3600c0ff00053cded9236be6401000000 dm-6 HPE,MSA 2050 SAN
size=13T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:3:0 sde 8:64  active ready running
| `- 2:0:3:0 sdl 8:176 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 1:0:5:0 sdg 8:96  active ready running
  `- 2:0:4:0 sdm 8:192 active ready running
36dcef8010056e1f12981675300000054 dm-5 HUAWEI,XSG1
size=1.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 1:0:0:1 sdb 8:16  active ready running
  |- 1:0:1:1 sdc 8:32  active ready running
  |- 1:0:2:1 sdd 8:48  active ready running
  |- 1:0:4:1 sdf 8:80  active ready running
  |- 2:0:0:1 sdh 8:112 active ready running
  |- 2:0:1:1 sdi 8:128 active ready running
  |- 2:0:2:1 sdj 8:144 active ready running
  `- 2:0:5:1 sdk 8:160 active ready running

Bash:

root@pve2:/etc/pve# pvdisplay -m /dev/mapper/3600c0ff00053cded9236be6401000000
  --- Physical volume ---
  PV Name               /dev/mapper/3600c0ff00053cded9236be6401000000
  VG Name               vg_ds1
  PV Size               12.73 TiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              3337857
  Free PE               3248247
  Allocated PE          89610
  PV UUID               wUtqQ1-o5PN-gau2-E2nO-jlfM-yJqR-Yw7Iqz
  
  --- Physical Segments ---
  Physical extent 0 to 54527:
    FREE
  Physical extent 54528 to 92933:
    Logical volume    /dev/vg_ds1/vm-114-disk-0.qcow2
    Logical extents    0 to 38405
  Physical extent 92934 to 113407:
    Logical volume    /dev/vg_ds1/vm-114-disk-1.qcow2
    Logical extents    0 to 20473
  Physical extent 113408 to 401417:
    FREE
  Physical extent 401418 to 406547:
    Logical volume    /dev/vg_ds1/vm-114-disk-1.qcow2
    Logical extents    20474 to 25603
  Physical extent 406548 to 432147:
    Logical volume    /dev/vg_ds1/vm-114-disk-2
    Logical extents    0 to 25599
  Physical extent 432148 to 3337856:
    FREE

Bash:

vmsrv200:~ #  b3sum /tst2/COMPLETE_DATA_BACKUP_databackup_3_1
7a8f4b5c54bc879275b7c56408603cc63adbb4118837f119e0ed8512c754d024  /tst2/COMPLETE_DATA_BACKUP_databackup_3_1
vmsrv200:~ #  cp /tst1/COMPLETE_DATA_BACKUP_databackup_3_1  /tst2/
vmsrv200:~ #  b3sum /tst2/COMPLETE_DATA_BACKUP_databackup_3_1
9a3c671bf904e29ae748512f2c0aff78fe15cf804f638358574b033258aad1cc  /tst2/COMPLETE_DATA_BACKUP_databackup_3_1

nothing in the VM-logs nor in the logs from the host
no sign of any FC-issues (the storages where former the vmware datastore - no problems in the past)
copy a second time the cecksum changes

any ideas?
not all files are broken .. (need to check more)
how debug that?

regards

Christian

fabian · Thursday at 12:58

compare the files and see where they differ (e.g., by running them through hexdump and comparing the dumps). try copying once more with `cp --sparse=never` and report back whether that copy is okay.

grisu68 · Thursday at 16:55

Hello,

after running lots of test with creating/deleting disks ...
(sorry .. lost the part in my logfiles .. and i cannot reproduce this on-demand - so no exactly docs )

if have created a zero-file (dd if=zero) and copied it to the disks ...
sometime ok ... but in a few cases:

vmsrv200:~ # hexdump -n 10 -b /tst1/file1
0000000 000 000 000 000 000 000 000 000 000 000
000000a
vmsrv200:~ # cp /tst1/file1 /tst2
vmsrv200:~ # hexdump -n 10 -b /tst2/file1
0000000 000 000 000 000 000 000 000 000 000 000
000000a
vmsrv200:~ # cp /tst1/file1 /tst3
vmsrv200:~ # hexdump -n 10 -b /tst3/file1
0000000 345 223 305 005 234 113 153 212 047 354
000000a

ok .... seems like a reuse of some uninitialzed parts of the disk/storage
and the filesystem in the VM cannot fix this during write

what do you think what could be wrong?

Chris

bbgeek17 · Thursday at 20:04

I'd recommend trying to reproduce this directly on the PVE host, bypassing the VM. Allocate a raw volume with pvesm alloc and test against the block device directly (or mkfs and mount it).

My suspicion is that LVM doesn't clear extents on delete (with saferemove 0) or before reuse. Combined with how your array handles zero/discard commands, this could lead to what you're seeing. With discard=on, PVE runs the disk with detect-zeroes=unmap. Zero writes from the guest are then passed down as write-zeroes requests with unmap allowed. On the host this becomes fallocate(PUNCH_HOLE) on the LV, which the kernel sends to the array as WRITE SAME (with the UNMAP bit if the array advertises LBPRZ). If the MSA acknowledges that without the range actually reading back as zeros, you get back whatever was on those extents before. That matches your zero-file test on /tst3. Note that this path only triggers when the guest actually writes zeros, so it may not explain every mismatch.

Based on your post, the only disk that behaves correctly (scsi4 on san1) is also the only one without discard=on,iothread=1,ssd=1. So right now there's no way to tell whether the array or the disk options are the cause.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

fabian · 2026-09-25T08:46:36+0200

we already have previous reports about HPE MSA (and Dell) appliances misbehaving in exactly this fashion - they advertise support for zeroing unmap, but only do it when the request matches some conditions, and simply ignore the zeroing part otherwise.

https://bugzilla.proxmox.com/show_bug.cgi?id=5754 (HPE MSA 1050)
https://bugzilla.proxmox.com/show_bug.cgi?id=7983 (Dell ME4)

data corruption on shared-lvm

grisu68

New Member

fabian

Proxmox Staff Member

grisu68

New Member

bbgeek17

Distinguished Member

fabian

Proxmox Staff Member

We value your privacy