create a shared storage with lvm

B

benoitc

Member
Dec 21, 2019
173
9
23
I read we can create an LVM storage over iscsi but how do you create the base volume and group? Should I create it on each machine? Any hint is welcome :)
 
Hi, can you please share a bit more details of your setup and desired goal?

Do you have a external storage box which you want to be able to access over iSCSI/LVM?

Or do you want to make the LVM from one node available on the other? If for HA or the like, it wouldn't achieve that, as once the node is offline all it's LVs cannot be accessed. For such a system it could be better to go with Ceph.

But that all depends on what your goal is :)
 
benoitc said:
I read we can create an LVM storage over iscsi but how do you create the base volume and group? Should I create it on each machine?
Click to expand...

After configuring the initiator on each node, you create the physical volume and the volume group on one node. All other nodes should see the volume (maybe after pvscan and vgscan but eventually after reboot). After confirming that all nodes see the vg, you can add it via the GUI (+ enable shared-checkbox) and be able to create VMs on it.
 
Hello Guys ,

Sorry for reopen an old issue, but I am facing a problem with proxmox 8, pve-manager/8.0.3/bbf3993334bfa916 (running kernel: 6.2.16-3-pve).

Any clue on below?

I do have an fc volume on top of HP 3PAR , mapped to a 3 node proxmox cluster.
I have installed multipath and enable it on all nodes.
created PV, VG and then created LVM from GUI from the first node.
When I tried to pvscan and vgscan the volume on the remaining 2 nodes , they simply cannot see it (multipath -ll shows nothing )

I am having the dmesg error, on both nodes, except the one that is working , apparently it cannot mount device/mapper volume group becaus is already in use on node1 :

343.579541] device-mapper: table: 253:8: multipath: error getting device (-EBUSY)
31603.802269 | 3PAR9TB: addmap [0 19327352832 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 4 1 8:16 1 8:32 1 8:48 1 8:64 1]
31603.804027 | libdevmapper: ioctl/libdm-iface.c(1980): device-mapper: reload ioctl on 3PAR9TB (253:8) failed: Device or resource busy

I do have on datacenter --> storage--> shared checkbox enabled for all nodes.
Below my multipath.conf, which is equal in all nodes

/etc/multipath.conf
defaults {
user_friendly_names yes
}

devices {
device {
vendor "3PARdata"
product "VV"
path_grouping_policy "group_by_prio"
path_checker "tur"
hardware_handler "1 alua"
prio "alua"
rr_weight "uniform"
no_path_retry "18"
failback "immediate"
}
}

blacklist {
wwid .*
}

blacklist_exceptions {
wwid "360002ac0000000000000008b000074ab"
}
multipaths {
multipath {
wwid "360002ac0000000000000008b000074ab"
alias 3PAR9TB
}
}

cat /etc/lvm/lvm.conf
config {
}
devices {
filter = [ "a|/dev/mapper/*|", "a|/dev/mapper/mpath*|","a|/dev/sda3|","r|/dev/zd*|", "r|/dev/sd*|","r/.*/" ]
}
allocation {
}
log {
}
backup {
}
shell {
}
global {
}
activation {
}
dmeventd {
}
devices {
global_filter=["r|/dev/zd.*|"]
}


Alua is enabled on 3par hosts.

Some outputs from nodes:

01:~# vgs
VG #PV #LV #SN Attr VSize VFree
GV_3PAR9TB 1 3 0 wz--n- 8.38t 7.84t
pve 1 3 0 wz--n- 278.36g 16.00g

00:~# vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- <135.70g 16.00g


02:~# vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- <135.70g 16.00g

root@TLT-DC-PXIRIS01:~# multipath -v3
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
360002ac0000000000000008b000074ab 1:0:0:1 sdb 8:16 50 undef undef 3PARdata,VV unknown
360002ac0000000000000008b000074ab 1:0:1:1 sdc 8:32 50 undef undef 3PARdata,VV unknown
360002ac0000000000000008b000074ab 2:0:0:1 sdd 8:48 50 undef undef 3PARdata,VV unknown
360002ac0000000000000008b000074ab 2:0:1:1 sde 8:64 50 undef undef 3PARdata,VV unknown
1380419.890898 | multipath-tools v0.9.4 (12/19, 2022)

root@TLT-DC-PXIRIS01:~# multipath -v3
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
360002ac0000000000000008b000074ab 1:0:0:1 sdb 8:16 50 undef undef 3PARdata,VV unknown
360002ac0000000000000008b000074ab 1:0:1:1 sdc 8:32 50 undef undef 3PARdata,VV unknown
360002ac0000000000000008b000074ab 2:0:0:1 sdd 8:48 50 undef undef 3PARdata,VV unknown
360002ac0000000000000008b000074ab 2:0:1:1 sde 8:64 50 undef undef 3PARdata,VV unknown
31603.794825 | multipath-tools v0.9.4 (12/19, 2022)

Thanks
 
jgarcia said:
When I tried to pvscan and vgscan the volume on the remaining 2 nodes , they simply cannot see it (multipath -ll shows nothing )
Click to expand...
Have you rescaned your bus or rebootet the machine? Please look at the scsi disks that are present on all nodes. You may want to install lsscsi and post the ouput from all three nodes (and please use CODE tags so that your output is readable).

Generally FC/iSCSI multipath setup is very straight forward and I've never seen your error message before. Looks interesting.
 
Hi,
Code: 
 filter = [ "a|/dev/mapper/*|", "a|/dev/mapper/mpath*|","a|/dev/sda3|","r|/dev/zd*|", "r|/dev/sd*|","r/.*/" ]
It might just be a formatting issue in the forum, but note that the LVM filters expect regular expressions, not wildcards. So to match any path starting with /dev/mapper/, you'd need to write /dev/mapper/.* (note the extra dot). Similarly with the other entries.

Can you try to refresh the initramfs before rebooting, to make sure it contains the custom LVM filters as well?
Code: 
update-initramfs -u -k all
 
Last edited:
Hi,

Working node

Code: 
root@PX01:~# lsblk
NAME                              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                 8:0    0 279.4G  0 disk 
├─sda1                              8:1    0  1007K  0 part 
├─sda2                              8:2    0     1G  0 part 
└─sda3                              8:3    0 278.4G  0 part 
  ├─pve-swap                      253:0    0     8G  0 lvm   [SWAP]
  ├─pve-root                      253:1    0  79.6G  0 lvm   /
  ├─pve-data_tmeta                253:2    0   1.7G  0 lvm   
  │ └─pve-data-tpool              253:4    0 171.3G  0 lvm   
  │   └─pve-data                  253:9    0 171.3G  1 lvm   
  └─pve-data_tdata                253:3    0 171.3G  0 lvm   
    └─pve-data-tpool              253:4    0 171.3G  0 lvm   
      └─pve-data                  253:9    0 171.3G  1 lvm   
sdb                                 8:16   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:10   0 174.7G  0 lvm   
sdc                                 8:32   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:10   0 174.7G  0 lvm   
sdd                                 8:48   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:10   0 174.7G  0 lvm   
sde                                 8:64   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:10   0 174.7G  0 lvm
    
root@TLT-DC-PX01:~# lsscsi
[0:0:0:0]    storage HP       P220i            5.42  -       
[0:1:0:0]    disk    HP       LOGICAL VOLUME   5.42  /dev/sda
[1:0:0:1]    disk    3PARdata VV               3213  /dev/sdb
[1:0:0:254]  enclosu 3PARdata SES              3213  -       
[1:0:1:1]    disk    3PARdata VV               3213  /dev/sdc
[1:0:1:254]  enclosu 3PARdata SES              3213  -       
[2:0:0:1]    disk    3PARdata VV               3213  /dev/sdd
[2:0:0:254]  enclosu 3PARdata SES              3213  -       
[2:0:1:1]    disk    3PARdata VV               3213  /dev/sde
[2:0:1:254]  enclosu 3PARdata SES              3213  -       
root@TLT-DC-PX01:~#

Nodes with errors
Code: 
PX00,PX02
Node
Code: 
PX00
is only seeing 2 paths , but this should not be an issue.

Code: 
root@PX00:~# lsblk
NAME                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                               8:0    0 558.9G  0 disk
├─sda1                            8:1    0  1007K  0 part
├─sda2                            8:2    0     1G  0 part
└─sda3                            8:3    0 557.9G  0 part
  ├─pve-swap                    253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                    253:1    0    96G  0 lvm  /
  ├─pve-data_tmeta              253:2    0   4.4G  0 lvm 
  │ └─pve-data                  253:4    0 429.1G  0 lvm 
  └─pve-data_tdata              253:3    0 429.1G  0 lvm 
    └─pve-data                  253:4    0 429.1G  0 lvm 
sdb                               8:16   0     9T  0 disk
└─sdb1                            8:17   0   8.4T  0 part
  ├─GV_3PAR9TB-vm--100--disk--1 253:5    0   280G  0 lvm 
  ├─GV_3PAR9TB-vm--101--disk--1 253:6    0 136.7G  0 lvm 
  └─GV_3PAR9TB-vm--101--disk--2 253:7    0 134.7G  0 lvm 
sdc                               8:32   0     9T  0 disk
└─sdc1                            8:33   0   8.4T  0 part
root@PX00:~#

root@PX00:~# lsscsi
[0:0:0:0]    storage HP       P220i            4.68  -       
[0:1:0:0]    disk    HP       LOGICAL VOLUME   4.68  /dev/sda
[1:0:0:1]    disk    3PARdata VV               3213  /dev/sdb
[1:0:0:254]  enclosu 3PARdata SES              3213  -       
[1:0:1:1]    disk    3PARdata VV               3213  /dev/sdc
[1:0:1:254]  enclosu 3PARdata SES              3213  -       
[2:0:0:254]  enclosu 3PARdata SES              3213  -       
[2:0:1:254]  enclosu 3PARdata SES              3213  -       
root@PX00:~#

root@PX02:~# lsblk
NAME                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                               8:0    0 136.7G  0 disk
├─sda1                            8:1    0  1007K  0 part
├─sda2                            8:2    0     1G  0 part
└─sda3                            8:3    0 135.7G  0 part
  ├─pve-swap                    253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                    253:1    0  43.9G  0 lvm  /
  ├─pve-data_tmeta              253:2    0     1G  0 lvm 
  │ └─pve-data                  253:4    0  65.8G  0 lvm 
  └─pve-data_tdata              253:3    0  65.8G  0 lvm 
    └─pve-data                  253:4    0  65.8G  0 lvm 
sdb                               8:16   0     9T  0 disk
└─sdb1                            8:17   0   8.4T  0 part
  ├─GV_3PAR9TB-vm--100--disk--1 253:5    0   280G  0 lvm 
  ├─GV_3PAR9TB-vm--101--disk--1 253:6    0 136.7G  0 lvm 
  └─GV_3PAR9TB-vm--101--disk--2 253:7    0 134.7G  0 lvm 
sdc                               8:32   0     9T  0 disk
└─sdc1                            8:33   0   8.4T  0 part
sdd                               8:48   0     9T  0 disk
└─sdd1                            8:49   0   8.4T  0 part
sde                               8:64   0     9T  0 disk
└─sde1                            8:65   0   8.4T  0 part
root@PX02:~#

root@PX02:~# lsscsi
[0:0:0:0]    storage HP       P410i            6.64  -       
[0:1:0:0]    disk    HP       LOGICAL VOLUME   6.64  /dev/sda
[1:0:0:1]    disk    3PARdata VV               3213  /dev/sdb
[1:0:0:254]  enclosu 3PARdata SES              3213  -       
[1:0:1:1]    disk    3PARdata VV               3213  /dev/sdc
[1:0:1:254]  enclosu 3PARdata SES              3213  -       
[2:0:0:1]    disk    3PARdata VV               3213  /dev/sdd
[2:0:0:254]  enclosu 3PARdata SES              3213  -       
[2:0:1:1]    disk    3PARdata VV               3213  /dev/sde
[2:0:1:254]  enclosu 3PARdata SES              3213  -       
root@PX02:~#

Thanks a lot guys, after refresh of initramfs as stated it worked.Everything seems fine now.
Nodes previously with errors, below:

Code: 
root@PX00:~# lsblk
NAME                              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                 8:0    0 558.9G  0 disk 
├─sda1                              8:1    0  1007K  0 part 
├─sda2                              8:2    0     1G  0 part 
└─sda3                              8:3    0 557.9G  0 part 
  ├─pve-swap                      253:0    0     8G  0 lvm   [SWAP]
  ├─pve-root                      253:1    0    96G  0 lvm   /
  ├─pve-data_tmeta                253:2    0   4.4G  0 lvm   
  │ └─pve-data                    253:4    0 429.1G  0 lvm   
  └─pve-data_tdata                253:3    0 429.1G  0 lvm   
    └─pve-data                    253:4    0 429.1G  0 lvm   
sdb                                 8:16   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:9    0 174.7G  0 lvm   
sdc                                 8:32   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:9    0 174.7G  0 lvm   
root@PX00:~# lsscsi
[0:0:0:0]    storage HP       P220i            4.68  -       
[0:1:0:0]    disk    HP       LOGICAL VOLUME   4.68  /dev/sda
[1:0:0:1]    disk    3PARdata VV               3213  /dev/sdb
[1:0:0:254]  enclosu 3PARdata SES              3213  -       
[1:0:1:1]    disk    3PARdata VV               3213  /dev/sdc
[1:0:1:254]  enclosu 3PARdata SES              3213  -       
[2:0:0:254]  enclosu 3PARdata SES              3213  -       
[2:0:1:254]  enclosu 3PARdata SES              3213  -       
root@PX00:~# vgs
  VG         #PV #LV #SN Attr   VSize    VFree
  GV_3PAR9TB   1   3   0 wz--n-    8.38t  7.77t
  pve          1   3   0 wz--n- <557.88g 16.00g
root@PX00:~#


root@PX02:~# lsblk
NAME                              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                 8:0    0 136.7G  0 disk 
├─sda1                              8:1    0  1007K  0 part 
├─sda2                              8:2    0     1G  0 part 
└─sda3                              8:3    0 135.7G  0 part 
  ├─pve-swap                      253:0    0     8G  0 lvm   [SWAP]
  ├─pve-root                      253:1    0  43.9G  0 lvm   /
  ├─pve-data_tmeta                253:2    0     1G  0 lvm   
  │ └─pve-data                    253:4    0  65.8G  0 lvm   
  └─pve-data_tdata                253:3    0  65.8G  0 lvm   
    └─pve-data                    253:4    0  65.8G  0 lvm   
sdb                                 8:16   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:9    0 174.7G  0 lvm   
sdc                                 8:32   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:9    0 174.7G  0 lvm   
sdd                                 8:48   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:9    0 174.7G  0 lvm   
sde                                 8:64   0     9T  0 disk 
└─3PAR9TB                         253:5    0     9T  0 mpath
  └─3PAR9TB-part1                 253:6    0   8.4T  0 part 
    ├─GV_3PAR9TB-vm--100--disk--1 253:7    0   280G  0 lvm   
    ├─GV_3PAR9TB-vm--101--disk--1 253:8    0 171.7G  0 lvm   
    └─GV_3PAR9TB-vm--101--disk--2 253:9    0 174.7G  0 lvm   
root@PX02:~# vgs
  VG         #PV #LV #SN Attr   VSize    VFree
  GV_3PAR9TB   1   3   0 wz--n-    8.38t  7.77t
  pve          1   3   0 wz--n- <135.70g 16.00g
root@PX02:~# lsscsi
[0:0:0:0]    storage HP       P410i            6.64  -       
[0:1:0:0]    disk    HP       LOGICAL VOLUME   6.64  /dev/sda
[1:0:0:1]    disk    3PARdata VV               3213  /dev/sdb
[1:0:0:254]  enclosu 3PARdata SES              3213  -       
[1:0:1:1]    disk    3PARdata VV               3213  /dev/sdc
[1:0:1:254]  enclosu 3PARdata SES              3213  -       
[2:0:0:1]    disk    3PARdata VV               3213  /dev/sdd
[2:0:0:254]  enclosu 3PARdata SES              3213  -       
[2:0:1:1]    disk    3PARdata VV               3213  /dev/sde
[2:0:1:254]  enclosu 3PARdata SES              3213  -       
root@PX02:~#

Once again, thanks to both of you guys.
 
You must log in or register to reply here.
Top Bottom