Reason found, LVM-error: "bcache_invalidate: block (0, 0) still dirty"

pp-king

New Member
Jul 13, 2024
6
0
1
Hi everyone,

I have to make a clarification:

Trying to create a CT/VM or a logical volume on my proxmox server does not work. lvcreate leads to the unclear error message

Code:
~# lvcreate -vvvv -n docker_home -L 30GB pm01_open

..
18:06:00.034945 label/label.c:1957  Error writing device /dev/sdb1 at 5632 length 1536.
01.051386 device/bcache.c:1374  WARNING: bcache_invalidate: block (0, 0) still dirty.
01.051424 format_text/format-text.c:965  Failed to write metadata to /dev/sdb1.
01.051430 metadata/metadata.c:3050  <backtrace>
01.051435 metadata/metadata.c:3059  Failed to write VG pm01_open.
..

Since I was able to create the LVM on my notebook, and simply plugged the drive (usb 3.1) from the notebook (Arch, working) to the server (Debian, not working), there must be a difference in LVM on arch and LVM on debian/proxmox.

Does anybody know, how to get rid of the "bcache_invalidate is dirty"

Thank you,
Peer


At the first moment I thought it was my mistake. Now, that I know the issue a little bit better, I renamed the title (fomerly: "Accidentially format a LVM partition with ext4"). Leaving the posting for its technical details.​


I wanted to consolidate my local IT environment using proxmox .. yesterday ;-) .. and made a pretty stupid mistake:

I created LVMs on two 14 TB HDDs and attached them via usb 3.1 as external drives. Since I had trouble getting them into the proxmox-storage I formated them newly from within proxmox (with ext4 ..) what made then unusable.

I knew, that LVM works with physical and logical volumes. But my idea was, proxmox would take care of the correct partititon type, change it, and afterwards format and access it.

Now the drives are still being shown as LVMs, but do always throw errors, when I try zu change them:

Bash:
~# pvcreate test -d /dev/sdb1

No device found for test.
WARNING: ext4 signature detected on /dev/sdb1 at offset 1080. Wipe it? [y/n]: y
Wiping ext4 signature on /dev/sdb1.
Error writing device /dev/sdb1 at 1080 length 2.
WARNING: bcache_invalidate: block (0, 0) still dirty.
Failed to wipe ext4 signature on /dev/sdb1.
1 existing signature left on the device.


If I wipe the disk:
Bash:
~# wipefs --all --force /dev/sdb
/dev/sdb: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sdb: 8 bytes were erased at offset 0xcbbbffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sdb: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa

But afterwards trying to change anything on the partition, i.e. delete it via cfdisk leads zu
( journalctl: )

Bash:
Jul 13 11:36:28 proxmox01 kernel: I/O error, dev sdc, sector 27344764888 op 0x1:(WRITE) flags 0x800 phys_seg 5prio class 0
Jul 13 11:36:28 proxmox01 kernel: Buffer I/O error on dev sdc, logical block 3418095611, lost async page write
Jul 13 11:36:28 proxmox01 kernel: Buffer I/O error on dev sdc, logical block 3418095612, lost async page write
Jul 13 11:36:28 proxmox01 kernel: Buffer I/O error on dev sdc, logical block 3418095613, lost async page write
Jul 13 11:36:28 proxmox01 kernel: Buffer I/O error on dev sdc, logical block 3418095614, lost async page write
Jul 13 11:36:28 proxmox01 kernel: Buffer I/O error on dev sdc, logical block 3418095615, lost async page write
Jul 13 11:46:46 proxmox01 pvedaemon[997]: <root@pam> successful auth for user 'root@pam'
[/code]

I also tried "chfdisk -z" and to "dd" the first 16 MB disk with random numbers. But no effect. Does anybody have an idea, how to get the disk working again?
Thanks :)
Peer

Here are some additional Information:

Bash:
# lsblk
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                  8:0    0 476.9G  0 disk
├─sda1               8:1    0  1007K  0 part
├─sda2               8:2    0     1G  0 part /boot/efi
└─sda3               8:3    0 475.9G  0 part
├─pve-swap       252:0    0     8G  0 lvm  [SWAP]
├─pve-root       252:1    0    96G  0 lvm  /
├─pve-data_tmeta 252:2    0   3.6G  0 lvm
│ └─pve-data     252:4    0 348.8G  0 lvm
└─pve-data_tdata 252:3    0 348.8G  0 lvm
└─pve-data     252:4    0 348.8G  0 lvm
sdb                  8:16   0  12.7T  0 disk
└─sdb1               8:17   0  12.7T  0 part
sdc                  8:32   0  12.7T  0 disk
└─sdc1               8:33   0  12.7T  0 part

~# fdisk -x /dev/sdb ; fdisk -x /dev/sdc
Bash:
Disk /dev/sdb: 12.73 TiB, 14000519643136 bytes, 27344764928 sectors
Disk model: MG07ACA14TE
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: <UUID>
First usable LBA: 2048
Last usable LBA: 27344764894
Alternative LBA: 27344764927
Partition entries starting LBA: 2
Allocated partition entries: 128
Partition entries ending LBA: 33

Device     Start         End     Sectors Type-UUID                            UUID                            Name Attrs
/dev/sdb1   2048 27344762879 27344760832 <UUID> <UUID>

Disk /dev/sdc: 12.73 TiB, 14000519643136 bytes, 27344764928 sectors
Disk model: MG07ACA14TE
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: <UUID>
First usable LBA: 2048
Last usable LBA: 27344764894
Alternative LBA: 27344764927
Partition entries starting LBA: 2
Allocated partition entries: 128
Partition entries ending LBA: 33

Device     Start         End     Sectors Type-UUID                            UUID                            Name Attrs
/dev/sdc1   2048 27344762879 27344760832 <UUID> <UUID>

Bash:
 
Last edited:
Maybe advanced logging on the pvcreate command helps to narrow down the problem:

Code:
29.967683 lvmcmdline.c:3140  Version: 2.03.16(2) (2022-05-18)
29.967699 lvmcmdline.c:3141  Parsing: pvcreate -vvvvv /dev/sdb1

[..]

29.968861 device/dev-cache.c:761  Found dev 8:16 /dev/disk/by-id/ata-TOSHIBA_MG07ACA14TE_Y3G0B07GUBWG - new alias.
29.968870 device/dev-cache.c:761  Found dev 8:17 /dev/disk/by-id/ata-TOSHIBA_MG07ACA14TE_Y3G0B07GUBWG-part1 - new alias.
29.968874 device/dev-cache.c:761  Found dev 8:32 /dev/disk/by-id/ata-TOSHIBA_MG07ACA14TE_Z320B08TUBWG - new alias.
29.968881 device/dev-cache.c:761  Found dev 8:33 /dev/disk/by-id/ata-TOSHIBA_MG07ACA14TE_Z320B08TUBWG-part1 - new alias.

[..]

29.973399 label/label.c:640  Scanning 1 devices for VG info
29.973405 label/label.c:568  open /dev/sdb1 ro di 0 fd 6
29.973460 label/label.c:679  Scanning submitted 1 reads
29.973849 label/label.c:713  Processing data from device /dev/sdb1 8:17 di 0
29.973863 device/dev-io.c:96  /dev/sdb1: using cached size 27344760832 sectors
29.973870 device/dev-mpath.c:469  /dev/sdb1: Device is a partition, using primary device sdb for mpath component detection
29.973897 device/dev-io.c:96  /dev/sdb1: using cached size 27344760832 sectors
29.974746 filters/filter-persistent.c:131  filter caching good /dev/sdb1
29.974754 label/label.c:398  /dev/sdb1: No lvm label detected
29.974761 label/label.c:748  Scanned devices: read errors 0 process errors 0 failed 0
29.974764 filters/filter-persistent.c:106  /dev/sdb1: filter cache using (cached good)
29.974766 toollib.c:5045  Checking /dev/sdb1 for pvcreate .
29.974768 toollib.c:5049  Check pvcreate arg /dev/sdb1 no PVID found
29.974772 toollib.c:5705  Rescanning and filtering device args with exclusive open
29.974774 label/label.c:640  Scanning 1 devices for VG info
29.974782 label/label.c:568  open /dev/sdb1 rwex di 0 fd 6
29.974826 label/label.c:679  Scanning submitted 1 reads
29.975204 label/label.c:713  Processing data from device /dev/sdb1 8:17 di 0
29.975220 device/dev-io.c:96  /dev/sdb1: using cached size 27344760832 sectors
29.975226 device/dev-mpath.c:469  /dev/sdb1: Device is a partition, using primary device sdb for mpath component detection
29.975256 device/dev-io.c:96  /dev/sdb1: using cached size 27344760832 sectors
29.976090 filters/filter-persistent.c:131  filter caching good /dev/sdb1
29.976098 label/label.c:398  /dev/sdb1: No lvm label detected
29.976100 label/label.c:748  Scanned devices: read errors 0 process errors 0 failed 0
29.976103 filters/filter-persistent.c:106  /dev/sdb1: filter cache using (cached good)
29.976105 toollib.c:5765  Wiping signatures on new PV /dev/sdb1.
29.976109 device_mapper/libdm-config.c:1085  allocation/use_blkid_wiping not found in config: defaulting to 1
29.980435 device/dev-type.c:925  Found existing signature on /dev/sdb1 at offset 1080: LABEL="(null)" UUID=<UUID> TYPE="ext4" USAGE="filesystem"

WARNING: ext4 signature detected on /dev/sdb1 at offset 1080. Wipe it? [y/n]: 14:38:32.024338 display/display.c:1022  Accepted input: [y]
32.024392 label/label.c:2054  Set last byte mixed block sizes physical 4096 logical 512 using 512
32.024402 device/bcache.c:229  Limit write at 0 len 131072 to len 1082 rounded to 1536
32.836628 label/label.c:2024  Error writing device /dev/sdb1 at 1080 length 2.
32.976061 device/bcache.c:1374  WARNING: bcache_invalidate: block (0, 0) still dirty.
32.976094 device/dev-type.c:941  Failed to wipe ext4 signature on /dev/sdb1.
32.976287 device/dev-type.c:1004  1 existing signature left on the device.
32.976375 toollib.c:5939  pvcreate: command failed for /dev/sdb1.
32.976382 toollib.c:5943  <backtrace>
32.976388 misc/lvm-flock.c:84  Unlocking /run/lock/lvm/P_global
32.976395 misc/lvm-flock.c:47  _undo_flock /run/lock/lvm/P_global
32.976421 device_mapper/libdm-config.c:1085  global/notify_dbus not found in config: defaulting to 1
32.976518 notify/lvmnotify.c:54  Nofify dbus at com.redhat.lvmdbus1.
32.976859 notify/lvmnotify.c:69  D-Bus notification failed: The name com.redhat.lvmdbus1 was not provided by any .service files
32.976884 cache/lvmcache.c:2589  Destroy lvmcache content
32.994129 lvmcmdline.c:3328  Completed: pvcreate -vvvvv /dev/sdb1
32.994407 cache/lvmcache.c:2589  Destroy lvmcache content
32.994424 metadata/vg.c:80  Freeing VG #orphans_lvm2 at 0x5db94d6ed4e0.
32.994466 activate/fs.c:492  Syncing device names
 
Hi,

I think, that I found the reason. And it belongs imho to the driver of the (cheap) SATA-hub (usb-id 174c:55aa):

On Debian/proxmox
I had problems to access SATA-hub at all and I had to blacklist the uas driver for this device, so that the storage has been accessed via the "usb-storage" driver (and the usb-storage driver may have problems to work with LVM?):

Code:
~# lsusb
..
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. ASM1051E..
..

~# lsusb -t
..
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
|__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 10000M
..

On Arch
I did not have to blacklist any driver. The usb device worked out of the box. And after testing/logging again, I found:

Code:
$ lsusb
..
Bus 002 Device 006: ID 174c:55aa ASMedia Technology Inc. ASM1051E ..
..

$ lsusb -t
..
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/2p, 10000M
|__ Port 002: Dev 006, If 0, Class=Mass Storage, Driver=uas, 5000M
..

So I assume, that this is a question of the different kernels in Debian and Arch, and that the newer kernel in Arch luckily support this device. Luckily because I learned, that the problems with the relevant chip are at least 6 years old.

Just for curiosity:
  1. Does anybody know or may show me, if and where I can verify this in the changelog of the kernel?
  2. Does anybody know, when the update will reach Debian and Debian with the Proxmox kernel?
Thank you and kind regards,
Peer
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!