[SOLVED] Start/Destroy: "Cannot activate LVs in VG while PVs appear on duplicate devices."

A

Adam Smith

Active Member
Feb 1, 2018
24
6
43
51
This is the same error that I posted here. However, the resolutions in that thread are not working. I've narrowed down the problem as a multipath issue, but I'm creating a new thread because this situation seems to be unique.

Node A and B were clustered and are running v6.4-13 and are identical hardware. They both have /dev/sda and /dev/sdb, as local devices, and /dev/sdc and /dev/sdd, multipathed and mapped to an iSCSI SAN . These were the same nodes with which I was having trouble setting up in the post above. Like I said in that post, I didn't wittingly set up multipath to the iSCSI SAN volume. It just so happened that addressing multipath solved that issue.

If removing multipath is the easier solution in this case, I would gladly do that, but I need some direction on how to gracefully back out of it.

Finally, here is the case.

Enter, Node C, a completely different model server I set up with v7.0-13 and then joined to the cluster. Multipath does not seem to work on it. I was able to migrate a VM to Node C, but starting in fails with the error Cannot activate LVs in VG while PVs appear on duplicate devices.. The same thing happens if I try to destroy the node.

Issuing lsblk shows the VM disks which reside on the iSCSI volume, while /dev/sdd is empty. Furthermore, neither device is mapped to the multipath wwid as they are on Node A and B.

Node C
Bash: 
sdc                      8:32   0     2T  0 disk
├─vms-vm--102--disk--1 253:0    0    40G  0 lvm
├─vms-vm--103--disk--0 253:1    0    40G  0 lvm
├─vms-vm--400--disk--0 253:2    0    16G  0 lvm
├─vms-vm--100--disk--0 253:3    0    40G  0 lvm
├─vms-vm--205--disk--0 253:4    0    25G  0 lvm
├─vms-vm--101--disk--0 253:5    0    40G  0 lvm
└─vms-vm--200--disk--0 253:6    0    40G  0 lvm
sdd                      8:48   0     2T  0 disk

Nodes A and B
Bash: 
sdc                                   8:32   0     2T  0 disk
└─36001a620000142313332304230313538 253:2    0     2T  0 mpath
  ├─vms-vm--102--disk--1            253:3    0    40G  0 lvm  
  ├─vms-vm--103--disk--0            253:4    0    40G  0 lvm  
  ├─vms-vm--400--disk--0            253:5    0    16G  0 lvm  
  ├─vms-vm--100--disk--0            253:6    0    40G  0 lvm  
  ├─vms-vm--205--disk--0            253:7    0    25G  0 lvm  
  ├─vms-vm--101--disk--0            253:8    0    40G  0 lvm  
  └─vms-vm--200--disk--0            253:9    0    40G  0 lvm  
sdd                                   8:48   0     2T  0 disk
└─36001a620000142313332304230313538 253:2    0     2T  0 mpath
  ├─vms-vm--102--disk--1            253:3    0    40G  0 lvm  
  ├─vms-vm--103--disk--0            253:4    0    40G  0 lvm  
  ├─vms-vm--400--disk--0            253:5    0    16G  0 lvm  
  ├─vms-vm--100--disk--0            253:6    0    40G  0 lvm  
  ├─vms-vm--205--disk--0            253:7    0    25G  0 lvm  
  ├─vms-vm--101--disk--0            253:8    0    40G  0 lvm  
  └─vms-vm--200--disk--0            253:9    0    40G  0 lvm

If I'm understanding the error correctly, since multipath is broken, the VMs cannot be found on both /dev/sdc /dev/sdd, and these actions fail.

Clearly multipath is broken on Node C, but again, if it is possible to safely disable multipath on Nodes A and B, this is perfectly acceptable. I still want to add two more nodes like C (with the intent of removing Nodes A and B) and I need to know the best way to proceed. Thank you in advance.
 
Last edited:
If you want to remove multipath, you have to make sure there aren't multiple paths advertised by your storage.

Please provide your multipath config (/etc/multipath.conf), the wwids file (/etc/multipath/wwids) and the output of multipath -ll of both nodes.
 
Is it odd that I don't have a /etc/multipath/multipath.conf file? On any node.

Nodes A and B:
Bash: 
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/36001a620000142313332304230313538/

36001a620000142313332304230313538 dm-2 Drobo,B800i
size=2.0T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 6:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:0:0 sdd 8:48 active ready running

Node C:
Bash: 
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:

[Edit: I did a multipath -W and removed an extraneous wwid]
 
Last edited:
It is /etc/multipath.conf ;)
Make sure the same config is available on Node C and if the WWID isn't added automatically, run multipath -a <WWID> on node C.
 
Thank you. I understand, only it isn't there either.
Bash: 
# cat /etc/multipath.conf
cat: /etc/multipath.conf: No such file or directory
Is this a symptom of a larger problem?

Results of multipath -T on Nodes A and B:
Bash: 
defaults {
        verbosity 2
        polling_interval 5
        max_polling_interval 20
        reassign_maps "no"
        multipath_dir "//lib/multipath"
        path_selector "service-time 0"
        path_grouping_policy "failover"
        uid_attribute "ID_SERIAL"
        prio "const"
        prio_args ""
        features "0"
        path_checker "tur"
        alias_prefix "mpath"
        failback "manual"
        rr_min_io 1000
        rr_min_io_rq 1
        max_fds "max"
        rr_weight "uniform"
        queue_without_daemon "no"
        flush_on_last_del "no"
        user_friendly_names "no"
        fast_io_fail_tmo 5
        bindings_file "/etc/multipath/bindings"
        wwids_file "/etc/multipath/wwids"
        prkeys_file "/etc/multipath/prkeys"
        log_checker_err always
        all_tg_pt "no"
        retain_attached_hw_handler "yes"
        detect_prio "yes"
        detect_checker "yes"
        force_sync "no"
        strict_timing "no"
        deferred_remove "no"
        config_dir "/etc/multipath/conf.d"
        delay_watch_checks "no"
        delay_wait_checks "no"
        marginal_path_err_sample_time "no"
        marginal_path_err_rate_threshold "no"
        marginal_path_err_recheck_gap_time "no"
        marginal_path_double_failed_time "no"
        find_multipaths "strict"
        uxsock_timeout 4000
        retrigger_tries 0
        retrigger_delay 10
        missing_uev_wait_timeout 30
        skip_kpartx "no"
        disable_changed_wwids "yes"
        remove_retries 0
        ghost_delay "no"
        find_multipaths_timeout -10
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|dcssblk)[0-9]"
        devnode "^(td|hd|vd)[a-z]"
        devnode "^cciss!c[0-9]d[0-9]*"
        device {
                vendor "SGI"
                product "Universal Xport"
        }
        device {
                vendor "^DGC"
                product "LUNZ"
        }
        device {
                vendor "EMC"
                product "LUNZ"
        }
        device {
                vendor "DELL"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "S/390"
        }
        device {
                vendor "(NETAPP|LSI|ENGENIO)"
                product "Universal Xport"
        }
        device {
                vendor "STK"
                product "Universal Xport"
        }
        device {
                vendor "SUN"
                product "Universal Xport"
        }
        device {
                vendor "(Intel|INTEL)"
                product "VTrak V-LUN"
        }
        device {
                vendor "Promise"
                product "VTrak V-LUN"
        }
        device {
                vendor "Promise"
                product "Vess V-LUN"
        }
}
blacklist_exceptions {
        property "(SCSI_IDENT_|ID_WWN)"
}
devices {
}
overrides {
}
multipaths {
        multipath {
                wwid "36001a620000142313332304230313538"
        }
}
 
mira said:
If you want to remove multipath, you have to make sure there aren't multiple paths advertised by your storage.
Click to expand...
I see what you're saying here. If I simply physically disconnect the second path, would this serve the same purpose or would it cause trouble with any running VMs? The reason I ask is because I don't think I have control over which paths the SAN advertises.
 
That depends on your setup.
But how about using that config as a base for all nodes?
 
I gave it a try and stumbled upon a resolution. I think this is what did it.
  1. created /etc/multipath.conf* on Node C and copied the config from Node B into it
  2. issued multipath with no arguments
  3. rebooted
After I issued multipath, it spit out this:
Bash: 
Oct 27 12:16:36 | 36001a620000142313332304230313538: addmap [0 4294967160 multipath 0 0 2 1 service-time 0 1 1 8:32 1 service-time 0 1 1 8:48 1]
create: 36001a620000142313332304230313538 undef Drobo,B800i
size=2.0T features='0' hwhandler='0' wp=undef
|-+- policy='service-time 0' prio=1 status=undef
| `- 7:0:0:0 sdc 8:32 undef ready running
`-+- policy='service-time 0' prio=1 status=undef
  `- 8:0:0:0 sdd 8:48 undef ready running
which looked hopeful. Still, lsblk looked funny:
Bash: 
NAME                                MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc                                   8:32   0     2T  0 disk
├─vms-vm--102--disk--1              253:0    0    40G  0 lvm
├─vms-vm--103--disk--0              253:1    0    40G  0 lvm
├─vms-vm--400--disk--0              253:2    0    16G  0 lvm
├─vms-vm--100--disk--0              253:3    0    40G  0 lvm
├─vms-vm--205--disk--0              253:4    0    25G  0 lvm
├─vms-vm--101--disk--0              253:5    0    40G  0 lvm
├─vms-vm--200--disk--0              253:6    0    40G  0 lvm
└─36001a620000142313332304230313538 253:7    0     2T  0 mpath
sdd                                   8:48   0     2T  0 disk
└─36001a620000142313332304230313538 253:7    0     2T  0 mpath
so I rebooted and it straightened out to match the other nodes:
Bash: 
sdc                                   8:32   0     2T  0 disk
└─36001a620000142313332304230313538 253:0    0     2T  0 mpath
  ├─vms-vm--102--disk--1            253:1    0    40G  0 lvm
  ├─vms-vm--103--disk--0            253:2    0    40G  0 lvm
  ├─vms-vm--400--disk--0            253:3    0    16G  0 lvm
  ├─vms-vm--100--disk--0            253:4    0    40G  0 lvm
  ├─vms-vm--205--disk--0            253:5    0    25G  0 lvm
  ├─vms-vm--101--disk--0            253:6    0    40G  0 lvm
  └─vms-vm--200--disk--0            253:7    0    40G  0 lvm
sdd                                   8:48   0     2T  0 disk
└─36001a620000142313332304230313538 253:0    0     2T  0 mpath
  ├─vms-vm--102--disk--1            253:1    0    40G  0 lvm
  ├─vms-vm--103--disk--0            253:2    0    40G  0 lvm
  ├─vms-vm--400--disk--0            253:3    0    16G  0 lvm
  ├─vms-vm--100--disk--0            253:4    0    40G  0 lvm
  ├─vms-vm--205--disk--0            253:5    0    25G  0 lvm
  ├─vms-vm--101--disk--0            253:6    0    40G  0 lvm
  └─vms-vm--200--disk--0            253:7    0    40G  0 lvm

Thank you for your guidance on this issue.

*Edit: I also had this happen with the next two nodes I added to the cluster. I learned that in addition to /etc/multipath.conf I also needed to copy /etc/multipath/wwids to each of the nodes. I did this with scp from the node with the config and wwids files: scp PATH-TO-FILE DEST-IP-ADDRESS:PATH-TO-DEST
 
Last edited:
  • Like
Reactions: mira
Thank you. It has been a while, but if I remember correctly, I was hesitant to reboot as a first solution because I had production VMs running from the storage in question.

At any rate, the real cause of my woes was the Drobo I was using could only be used in iSCSI mode on both of its ethernet ports, so multipath was a must. I was unsatisfied with this arrangement, because I was trying to use one of the ports for management-only on my primary subnet. I did not want storage to be flooding that subnet with traffic. Also, I was being bothered by the advertisement each time I opened Disk Manager on Windows. Disk Manager would offer to initialize the iSCSI volume it was detecting when all I wanted to do was manage the Drobo with the Drobo console. So, no mo Drobo.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back