Device mismatch detected for IBM-SAS-900G-PROD-03/vm-11201003-disk-0 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03

D

Dennis Ayala

Well-Known Member
Jul 20, 2018
43
4
48
53
I have a 3 node cluster with an Fibre Channel Storage configured as Shared LVM. It was installed using Proxmox 7 and was working great. Once we upgraded to Proxmox 8 I started to notice this messages on some of the virtual machines.

If I run a pvscan on a host #1:

Code: 
# pvscan
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-11201003-disk-0 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-11201003-disk-1 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-20051062-disk-0 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-23403001-disk-0 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-23403001-disk-1 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  PV /dev/mapper/IBM-SAS-900G-PROD-03   VG IBM-SAS-900G-PROD-03   lvm2 [3.27 TiB / 971.91 GiB free]
  PV /dev/mapper/IBM-SAS-900G-PROD-01   VG IBM-SAS-900G-PROD-01   lvm2 [3.27 TiB / <1.34 TiB free]
  PV /dev/mapper/IBM-SAS-900G-PROD-07   VG IBM-SAS-900G-PROD-07   lvm2 [3.27 TiB / <701.45 GiB free]
  PV /dev/mapper/IBM-SAS-900G-PROD-05   VG IBM-SAS-900G-PROD-05   lvm2 [3.27 TiB / <925.97 GiB free]
  PV /dev/sda3                          VG pve                    lvm2 [<135.47 GiB / 16.00 GiB free]
  Total: 5 [13.22 TiB] / in use: 5 [13.22 TiB] / in no VG: 0 [0   ]

All other nodes are fine, only this node shows this message.

When I run lvscan on the same host:

Code: 
# lvscan
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-11201003-disk-0 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-11201003-disk-1 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-20051062-disk-0 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-23403001-disk-0 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  WARNING: Device mismatch detected for IBM-SAS-900G-PROD-03/vm-23403001-disk-1 which is accessing /dev/sdq instead of /dev/mapper/IBM-SAS-900G-PROD-03.
  inactive          '/dev/IBM-SAS-900G-PROD-03/vm-1061-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-03/vm-11201003-disk-0' [<136.13 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-03/vm-11201003-disk-1' [<583.41 GiB] inherit
  inactive          '/dev/IBM-SAS-900G-PROD-03/vm-1020-disk-1' [<1.27 TiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-03/vm-20051062-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-03/vm-23403001-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-03/vm-23403001-disk-1' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-1051-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-40201017-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-40201017-disk-1' [200.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-20051063-disk-1' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-20051064-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-54003012-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-54003013-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-54001021-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-54001022-disk-0' [<68.25 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-54001022-disk-1' [68.24 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-54003010-disk-0' [<74.50 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-40201021-disk-0' [100.00 GiB] inherit
  inactive          '/dev/IBM-SAS-900G-PROD-01/vm-1016-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-22751033-disk-0' [60.00 GiB] inherit
  inactive          '/dev/IBM-SAS-900G-PROD-01/vm-22751031-disk-0' [60.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-22751032-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-4221002-disk-0' [250.00 GiB] inherit
  inactive          '/dev/IBM-SAS-900G-PROD-01/vm-1020-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-3002-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-5010001-disk-0' [4.00 MiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-01/vm-5010001-disk-1' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-40201014-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-20051063-disk-1' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-81101062-disk-0' [250.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-40201018-disk-1' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-40201020-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-40201020-disk-1' [1.46 TiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-1020-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-5010002-disk-0' [4.00 MiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-5010002-disk-1' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-07/vm-5010002-disk-2' [300.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1001-disk-0' [200.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1001-disk-1' [<558.38 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1010-disk-1' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1011-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1012-disk-0' [30.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1012-disk-1' [200.00 GiB] inherit
  inactive          '/dev/IBM-SAS-900G-PROD-05/vm-1016-disk-1' [500.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1015-disk-0' [150.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1015-disk-1' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-1013-disk-0' [100.00 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-11201002-disk-0' [67.73 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-11201002-disk-1' [279.37 GiB] inherit
  ACTIVE            '/dev/IBM-SAS-900G-PROD-05/vm-20051063-disk-0' [80.00 GiB] inherit
  ACTIVE            '/dev/pve/data' [<65.60 GiB] inherit
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [43.87 GiB] inherit

lvm.conf device global filter:

Code: 
devices {
     # added by pve-manager to avoid scanning ZFS zvols
     global_filter=["r|/dev/zd.*|"]
 }
 
I've moved the thread from the Proxmox Backup Server forum to the Proxmox VE forum.

I assume you have set up multipath -- can you post the output of multipath -ll and the contents of /etc/multipath/wwids? It looks like LVM uses one of the multipath components (/dev/sdq) instead of the multipath device (/dev/mapper/IBM-SAS-900G-PROD-03). This usually happens because LVM scans for physical volumes already during early boot before the multipath device has been set up, and decides to use one of the multipath components. To get rid of the messages for the current boot, you can try running vgchange --refresh VG_NAME. But most likely, the messages will appear again after the next boot. To get rid of the warnings permanently, you can try installing the package multipath-tools-boot [1]. With this package installed, the multipath configuration /etc/multipath.conf and /etc/multipath/wwids are made available in the initramfs and thus the early boot process. Thus, LVM will be able to tell that the block devices are multipath components, ignore them and use the multipath device instead when it's available.

Side note: An alternative workaround would be define a global_filter that excludes the multipath components. These filters should preferably use symlinks to identfy the disks (e.g. /dev/disk/by-id/...). However, the LVM version currently shipped with Debian Bookworm / PVE 8 (2.03.16-3) has a regression due to which filters with symlinks do not always work, see [2] for more information. So currently the preferred workaround would be to install multipath-tools-boot as described above.

If you try this workaround, please let me know it fixes the problem for you.

[1] https://packages.debian.org/bookworm/multipath-tools-boot
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028541#36
 
  • Like
Reactions: tell and ptsbkhairul
While running multipath -ll I discovered a typo in the blacklist segment. I had wwids.* key instead of wwid.*.

I fixed it and will see if it fixes the issue. If not, I will post more info.

Thank you,

Dennis
 
  • Like
Reactions: fweber
This happened to us yesterday after we upgraded to 8.2 from 8.1.
Luckily this thread gave us the correct path to follow right away.
With the command "vgchange --refresh VG_NAME" we realized that "Device mismatch detected" errors disappeared.
So, we immediately installed the multipath-tools-boot package and the problem seems solved.
We have BladeCenter C7000 with FiberChannel storage (EMC VNX5200) with Shared lvm.
Thank you guys
 
Last edited:
  • Like
Reactions: fweber
ermurenz said:
This happened to us yesterday after we upgraded to 8.2 from 8.1.
Luckily this thread gave us the correct path to follow right away.
With the command "vgchange --refresh VG_NAME" we realized that "Device mismatch detected" errors disappeared.
So, we immediately installed the multipath-tools-boot package and the problem seems solved.
We have BladeCenter C7000 with FiberChannel storage (EMC VNX5200) with Shared lvm.
Thank you guys
Click to expand...
Hello:

I am glad it helped.

Regards,

Dennis
 
  • Like
Reactions: ermurenz
Hi everyone, we have similar issue in our environment. After running the "vgchange --refresh VG_NAME", it works fine, however, after every reboot the "vgchange --refresh VG_NAME" string has to be run again as the issue reoccurs -- seems a workaround. Any ideas out there for permanent fix?
 
Karifa said:
Hi everyone, we have similar issue in our environment. After running the "vgchange --refresh VG_NAME", it works fine, however, after every reboot the "vgchange --refresh VG_NAME" string has to be run again as the issue reoccurs -- seems a workaround. Any ideas out there for permanent fix?
Click to expand...
Hi, can you try whether the installing multipath-tools-boot, as described here [1], fixes the issue for you?

[1] https://forum.proxmox.com/threads/140416/post-629413
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back