elastic vm showing bad block bitmap checksum on ext4

D

dme31

New Member
Oct 14, 2026
3
0
1
Hi !

I am requesting help !

My setup is :

Dell ME5 with 2 controllers ( mainly composed of 7200 rpm 16TB hard disk and 3.8TB ssd)

2 x 10GB switches aruba 8360-x48xtc v2 dédicated for the iscsi
3 x proxmox nodes 9.2.11 running on Hpe proliant dl360 gen11 server
Each with 2 x 10Gb nic 10Base-T BCM57416 with bnxt_en driver dédicated for the iscsi

Kernel Version 7.0.14-12-pve (2026-08-11T11:05Z)
multipath fully functionnal with all path running
Thick shared LVM built on top of the lun.
no iscsi timeout
no warnings on the san
firmware of the dell me5 up to date
firmwares of the dell me5 hard disk drive and ssd up to date
firmware of the HPe servers up to date.

The problem is that we have an Elastic cluster composed of 5 Hot nodes who are throwing the following error :

EXT4-fs error (device dm-7): ext4_validate_block_bitmap:423: comm elastic..][T#6]: bg 19130: bad block bitmap checksum
EXT4-fs (dm-7): Delayed block allocation failed for inode 76285244 at logical offset 8192 with max blocks 74 with error 74
EXT4-fs (dm-7): This should not happen!! Data will be lost

all five nodes are affected, and some others vms that are not running elastic but some others are totaly fine.
every time it is happening, we stop the elastic service, unmount the partition, run a fsck which clean the Filesystem and reboot the system.
the cluster goes green again until the same kind of error happens 2 or 3 days later and then we go again....

We have checked everything and disabled the offloading features of the bnxt driver without luck.

if you have any clue to help me sort out this issue i will be more than happy.

Thanks in advance for your advices
 
Last edited:
Hi @dme31 , welcome to the forum.

Your situation might be related to this thread: PVE 9.2.x, and shared thick LVM on multipath, with guests reading back data different from what was written and nothing in the host or SAN logs. The summary of that thread is that the SAN lies about writes, leading to data corruption on the host.

There are a few things you can do to help with troubleshooting:
- post qm config for an affected VM and your storage.cfg entry (disk format, discard, aio, snapshot-as-volume-chain)
- the guest mount options and whether fstrim runs?

As a test, try disabling discard on those disks, and if possible reproduce on a scratch disk on the same storage with fio --verify=crc32c.

Keep in mind that fsck only repairs metadata and the “data will be lost” message means real data is affected, so make sure Elastic has replicas or snapshots elsewhere.

Of course if this is affecting your production the best step is to involve your vendors (Proxmox, Dell, etc).

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hi !

Thanks for your help :)

Here is the qm config file for one vm : The corrupted file system is on the scsi1

agent: 1
balloon: 0
boot: order=scsi0;net0
cores: 16
cpu: x86-64-v4
kvm: 1
memory: 65536
meta: creation-qemu=9.2.0,ctime=1769786652
name: host-05
net0: virtio=BC:24:11:2D:8D:7B,bridge=vmbr0
numa: 1
onboot: 0
ostype: l26
scsi0: san-vol-2:vm-525-disk-0,cache=none,discard=on,iothread=1,replicate=0,size=25G
scsi1: san-vol-2:vm-525-disk-1,backup=0,cache=none,discard=on,iothread=1,replicate=0,size=3000G
scsihw: virtio-scsi-single
smbios1: uuid=58a7a629-db47-470a-a095-0fe84514f6b9
sockets: 2
vmgenid: 15a0b7b2-78da-4df8-8fbd-b50bda3f855a


here is the proxmox storage.cfg

dir: local
path /var/lib/vz
content iso
prune-backups keep-all=1
shared 0

lvm: san-vol-1
vgname vg_vol_1
content images
saferemove 0
shared 1
snapshot-as-volume-chain 1

lvm: san-vol-2
vgname vg_vol_2
content images
saferemove 0
shared 1
snapshot-as-volume-chain 1

lvm: san-vol-3
vgname vg_vol_3
content images
saferemove 0
shared 1
snapshot-as-volume-chain 1


the guest fstab mount options :

/dev/mapper/vgdata /DATA ext4 defaults 0 2


status of fstrim.timer :

● fstrim.timer - Discard unused filesystem blocks once a week
Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; preset: enabled)
Active: active (waiting) since Wed 2026-10-14 15:23:37 CEST; 17h ago
Invocation: 59854706999740399a9237a7b9918b74
Trigger: Mon 2026-10-19 01:34:16 CEST; 3 days left
Triggers: ● fstrim.service
Docs: man:fstrim

oct. 14 15:23:37 host-05 systemd[1]: Started fstrim.timer - Discard unused filesystem blocks once a week.

We reached the dell support and they see noting related to our problem

best regards !
 
Last edited:
dme31 said:
We reached the dell support and they see noting related to our problem
Click to expand...
This is not unexpected. The next steps would be to try the suggested testing and/or reach out to Proxmox support.
bbgeek17 said:
As a test, try disabling discard on those disks, and if possible reproduce on a scratch disk on the same storage with fio --verify=crc32c.
Click to expand...

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
This answer is AI assisted:

Everything below writes to the target, so it must be a scratch disk, not the Elastic one.

Step 1: Raw-device write and verify. Add a scratch virtual disk to a test VM on the same storage (san-vol-2), with the same options as the Elastic disks (discard=on, iothread=1, cache=none, virtio-scsi-single). Then run this in the guest, with /dev/sdX as the scratch disk:
Code: 
fio --name=crc-test --filename=/dev/sdX --direct=1 --ioengine=libaio \
    --rw=randwrite --bs=4k --iodepth=32 --numjobs=4 \
    --size=50G --offset_increment=50G \
    --randseed=1234 \
    --verify=crc32c --verify_fatal=1 --verify_dump=1 \
    --group_reporting
Each job gets its own 50G region, writes a checksummed pattern, then reads it back and verifies. verify_fatal stops at the first mismatch, and verify_dump saves the bad data for inspection.

Step 2: Delayed re-verify. The corruption shows up 2 to 3 days later, so an immediate read-back may pass. After a run completes, wait a few hours (or a day or two), run fstrim on other volumes on the same storage, and then re-read the same data with the same parameters:
Code: 
fio --name=crc-test --filename=/dev/sdX --direct=1 --ioengine=libaio \
    --rw=randwrite --bs=4k --iodepth=32 --numjobs=4 \
    --size=50G --offset_increment=50G \
    --randseed=1234 \
    --verify=crc32c --verify_only --verify_fatal=1 --verify_dump=1 \
    --group_reporting

  • run the test on several VMs at once, ideally spanning all three nodes
  • repeat it with discard disabled on the disk and compare results
  • if the raw device never fails, put ext4 on the scratch disk, run fio against files on it in a loop, and run fstrim on it periodically
Host level. A scratch LV tested with the same fio command directly on a PVE node (no VM involved) tells him whether the problem is in the guest stack or below it, in multipath, the SAN, or LVM.

Or you can read the other thread and associated bugzilla entry and implement the suggestions mentioned there.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
You must log in or register to reply here.
Top Bottom
Back