[SOLVED] Start/Destroy: "Cannot activate LVs in VG while PVs appear on duplicate devices."

A

Adam Smith

Active Member
Feb 1, 2018
24
6
43
53
This is the same error that I posted here. However, the resolutions in that thread are not working. I've narrowed down the problem as a multipath issue, but I'm creating a new thread because this situation seems to be unique.

Node A and B were clustered and are running v6.4-13 and are identical hardware. They both have /dev/sda and /dev/sdb, as local devices, and /dev/sdc and /dev/sdd, multipathed and mapped to an iSCSI SAN . These were the same nodes with which I was having trouble setting up in the post above. Like I said in that post, I didn't wittingly set up multipath to the iSCSI SAN volume. It just so happened that addressing multipath solved that issue.

If removing multipath is the easier solution in this case, I would gladly do that, but I need some direction on how to gracefully back out of it.

Finally, here is the case.

Enter, Node C, a completely different model server I set up with v7.0-13 and then joined to the cluster. Multipath does not seem to work on it. I was able to migrate a VM to Node C, but starting in fails with the error Cannot activate LVs in VG while PVs appear on duplicate devices.. The same thing happens if I try to destroy the node.

Issuing lsblk shows the VM disks which reside on the iSCSI volume, while /dev/sdd is empty. Furthermore, neither device is mapped to the multipath wwid as they are on Node A and B.

Node C
Bash: 
sdc                      8:32   0     2T  0 disk
├─vms-vm--102--disk--1 253:0    0    40G  0 lvm
├─vms-vm--103--disk--0 253:1    0    40G  0 lvm
├─vms-vm--400--disk--0 253:2    0    16G  0 lvm
├─vms-vm--100--disk--0 253:3    0    40G  0 lvm
├─vms-vm--205--disk--0 253:4    0    25G  0 lvm
├─vms-vm--101--disk--0 253:5    0    40G  0 lvm
└─vms-vm--200--disk--0 253:6    0    40G  0 lvm
sdd                      8:48   0     2T  0 disk

Nodes A and B
Bash: 
sdc                                   8:32   0     2T  0 disk
└─36001a620000142313332304230313538 253:2    0     2T  0 mpath
  ├─vms-vm--102--disk--1            253:3    0    40G  0 lvm  
  ├─vms-vm--103--disk--0            253:4    0    40G  0 lvm  
  ├─vms-vm--400--disk--0            253:5    0    16G  0 lvm  
  ├─vms-vm--100--disk--0            253:6    0    40G  0 lvm  
  ├─vms-vm--205--disk--0            253:7    0    25G  0 lvm  
  ├─vms-vm--101--disk--0            253:8    0    40G  0 lvm  
  └─vms-vm--200--disk--0            253:9    0    40G  0 lvm  
sdd                                   8:48   0     2T  0 disk
└─36001a620000142313332304230313538 253:2    0     2T  0 mpath
  ├─vms-vm--102--disk--1            253:3    0    40G  0 lvm  
  ├─vms-vm--103--disk--0            253:4    0    40G  0 lvm  
  ├─vms-vm--400--disk--0            253:5    0    16G  0 lvm  
  ├─vms-vm--100--disk--0            253:6    0    40G  0 lvm  
  ├─vms-vm--205--disk--0            253:7    0    25G  0 lvm  
  ├─vms-vm--101--disk--0            253:8    0    40G  0 lvm  
  └─vms-vm--200--disk--0            253:9    0    40G  0 lvm

If I'm understanding the error correctly, since multipath is broken, the VMs cannot be found on both /dev/sdc /dev/sdd, and these actions fail.

Clearly multipath is broken on Node C, but again, if it is possible to safely disable multipath on Nodes A and B, this is perfectly acceptable. I still want to add two more nodes like C (with the intent of removing Nodes A and B) and I need to know the best way to proceed. Thank you in advance.
 
Last edited:
If you want to remove multipath, you have to make sure there aren't multiple paths advertised by your storage.

Please provide your multipath config (/etc/multipath.conf), the wwids file (/etc/multipath/wwids) and the output of multipath -ll of both nodes.
 
Is it odd that I don't have a /etc/multipath/multipath.conf file? On any node.

Nodes A and B:
Bash: 
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/36001a620000142313332304230313538/

36001a620000142313332304230313538 dm-2 Drobo,B800i
size=2.0T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 6:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 7:0:0:0 sdd 8:48 active ready running

Node C:
Bash: 
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:

[Edit: I did a multipath -W and removed an extraneous wwid]
 
Last edited:
It is /etc/multipath.conf ;)
Make sure the same config is available on Node C and if the WWID isn't added automatically, run multipath -a <WWID> on node C.
 
Thank you. I understand, only it isn't there either.
Bash: 
# cat /etc/multipath.conf
cat: /etc/multipath.conf: No such file or directory
Is this a symptom of a larger problem?

Results of multipath -T on Nodes A and B:
Bash: 
defaults {
        verbosity 2
        polling_interval 5
        max_polling_interval 20
        reassign_maps "no"
        multipath_dir "//lib/multipath"
        path_selector "service-time 0"
        path_grouping_policy "failover"
        uid_attribute "ID_SERIAL"
        prio "const"
        prio_args ""
        features "0"
        path_checker "tur"
        alias_prefix "mpath"
        failback "manual"
        rr_min_io 1000
        rr_min_io_rq 1
        max_fds "max"
        rr_weight "uniform"
        queue_without_daemon "no"
        flush_on_last_del "no"
        user_friendly_names "no"
        fast_io_fail_tmo 5
        bindings_file "/etc/multipath/bindings"
        wwids_file "/etc/multipath/wwids"
        prkeys_file "/etc/multipath/prkeys"
        log_checker_err always
        all_tg_pt "no"
        retain_attached_hw_handler "yes"
        detect_prio "yes"
        detect_checker "yes"
        force_sync "no"
        strict_timing "no"
        deferred_remove "no"
        config_dir "/etc/multipath/conf.d"
        delay_watch_checks "no"
        delay_wait_checks "no"
        marginal_path_err_sample_time "no"
        marginal_path_err_rate_threshold "no"
        marginal_path_err_recheck_gap_time "no"
        marginal_path_double_failed_time "no"
        find_multipaths "strict"
        uxsock_timeout 4000
        retrigger_tries 0
        retrigger_delay 10
        missing_uev_wait_timeout 30
        skip_kpartx "no"
        disable_changed_wwids "yes"
        remove_retries 0
        ghost_delay "no"
        find_multipaths_timeout -10
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|dcssblk)[0-9]"
        devnode "^(td|hd|vd)[a-z]"
        devnode "^cciss!c[0-9]d[0-9]*"
        device {
                vendor "SGI"
                product "Universal Xport"
        }
        device {
                vendor "^DGC"
                product "LUNZ"
        }
        device {
                vendor "EMC"
                product "LUNZ"
        }
        device {
                vendor "DELL"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "S/390"
        }
        device {
                vendor "(NETAPP|LSI|ENGENIO)"
                product "Universal Xport"
        }
        device {
                vendor "STK"
                product "Universal Xport"
        }
        device {
                vendor "SUN"
                product "Universal Xport"
        }
        device {
                vendor "(Intel|INTEL)"
                product "VTrak V-LUN"
        }
        device {
                vendor "Promise"
                product "VTrak V-LUN"
        }
        device {
                vendor "Promise"
                product "Vess V-LUN"
        }
}
blacklist_exceptions {
        property "(SCSI_IDENT_|ID_WWN)"
}
devices {
}
overrides {
}
multipaths {
        multipath {
                wwid "36001a620000142313332304230313538"
        }
}
 
mira said:
If you want to remove multipath, you have to make sure there aren't multiple paths advertised by your storage.
Click to expand...
I see what you're saying here. If I simply physically disconnect the second path, would this serve the same purpose or would it cause trouble with any running VMs? The reason I ask is because I don't think I have control over which paths the SAN advertises.
 
That depends on your setup.
But how about using that config as a base for all nodes?
 
I gave it a try and stumbled upon a resolution. I think this is what did it.
  1. created /etc/multipath.conf* on Node C and copied the config from Node B into it
  2. issued multipath with no arguments
  3. rebooted
After I issued multipath, it spit out this:
Bash: 
Oct 27 12:16:36 | 36001a620000142313332304230313538: addmap [0 4294967160 multipath 0 0 2 1 service-time 0 1 1 8:32 1 service-time 0 1 1 8:48 1]
create: 36001a620000142313332304230313538 undef Drobo,B800i
size=2.0T features='0' hwhandler='0' wp=undef
|-+- policy='service-time 0' prio=1 status=undef
| `- 7:0:0:0 sdc 8:32 undef ready running
`-+- policy='service-time 0' prio=1 status=undef
  `- 8:0:0:0 sdd 8:48 undef ready running
which looked hopeful. Still, lsblk looked funny:
Bash: 
NAME                                MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdc                                   8:32   0     2T  0 disk
├─vms-vm--102--disk--1              253:0    0    40G  0 lvm
├─vms-vm--103--disk--0              253:1    0    40G  0 lvm
├─vms-vm--400--disk--0              253:2    0    16G  0 lvm
├─vms-vm--100--disk--0              253:3    0    40G  0 lvm
├─vms-vm--205--disk--0              253:4    0    25G  0 lvm
├─vms-vm--101--disk--0              253:5    0    40G  0 lvm
├─vms-vm--200--disk--0              253:6    0    40G  0 lvm
└─36001a620000142313332304230313538 253:7    0     2T  0 mpath
sdd                                   8:48   0     2T  0 disk
└─36001a620000142313332304230313538 253:7    0     2T  0 mpath
so I rebooted and it straightened out to match the other nodes:
Bash: 
sdc                                   8:32   0     2T  0 disk
└─36001a620000142313332304230313538 253:0    0     2T  0 mpath
  ├─vms-vm--102--disk--1            253:1    0    40G  0 lvm
  ├─vms-vm--103--disk--0            253:2    0    40G  0 lvm
  ├─vms-vm--400--disk--0            253:3    0    16G  0 lvm
  ├─vms-vm--100--disk--0            253:4    0    40G  0 lvm
  ├─vms-vm--205--disk--0            253:5    0    25G  0 lvm
  ├─vms-vm--101--disk--0            253:6    0    40G  0 lvm
  └─vms-vm--200--disk--0            253:7    0    40G  0 lvm
sdd                                   8:48   0     2T  0 disk
└─36001a620000142313332304230313538 253:0    0     2T  0 mpath
  ├─vms-vm--102--disk--1            253:1    0    40G  0 lvm
  ├─vms-vm--103--disk--0            253:2    0    40G  0 lvm
  ├─vms-vm--400--disk--0            253:3    0    16G  0 lvm
  ├─vms-vm--100--disk--0            253:4    0    40G  0 lvm
  ├─vms-vm--205--disk--0            253:5    0    25G  0 lvm
  ├─vms-vm--101--disk--0            253:6    0    40G  0 lvm
  └─vms-vm--200--disk--0            253:7    0    40G  0 lvm

Thank you for your guidance on this issue.

*Edit: I also had this happen with the next two nodes I added to the cluster. I learned that in addition to /etc/multipath.conf I also needed to copy /etc/multipath/wwids to each of the nodes. I did this with scp from the node with the config and wwids files: scp PATH-TO-FILE DEST-IP-ADDRESS:PATH-TO-DEST
 
Last edited:
  • Like
Reactions: mira
Thank you. It has been a while, but if I remember correctly, I was hesitant to reboot as a first solution because I had production VMs running from the storage in question.

At any rate, the real cause of my woes was the Drobo I was using could only be used in iSCSI mode on both of its ethernet ports, so multipath was a must. I was unsatisfied with this arrangement, because I was trying to use one of the ports for management-only on my primary subnet. I did not want storage to be flooding that subnet with traffic. Also, I was being bothered by the advertisement each time I opened Disk Manager on Windows. Disk Manager would offer to initialize the iSCSI volume it was detecting when all I wanted to do was manage the Drobo with the Drobo console. So, no mo Drobo.
 
You must log in or register to reply here.
Top Bottom