Cluster and IBM storage

Kirill · Sep 24, 2014

Good day.
I have 2 servers IBM X3550 M3 with HBA SAS adapter and IBM System Storage DH3524 SAS. I have installed Proxmox 3.3 on it. When I try to connect the storage system, it sees only one node.

pvecm status

Code:

root@pve1:~# pvecm status
Version: 6.2.0
Config Version: 2
Cluster Name: pve-cluster
Cluster Id: 23476
Cluster Member: Yes
Cluster Generation: 20
Membership state: Cluster-Member
Nodes: 2
Expected votes: 2
Total votes: 2
Node votes: 1
Quorum: 2
Active subsystems: 6
Flags:
Ports Bound: 0 11
Node name: pve1
Node ID: 1
Multicast addresses: 239.192.91.16
Node addresses: 192.168.199.200

fdisk -l on pve1

Code:

root@pve1:~# fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 146.0 GB, 145999527936 bytes
255 heads, 63 sectors/track, 17750 cylinders, total 285155328 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1   285155327   142577663+  ee  GPT

Disk /dev/sdb: 1796.4 GB, 1796370071552 bytes
57 heads, 28 sectors/track, 2198330 cylinders, total 3508535296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xa14806fa

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3508535295  1754266624   83  Linux

Disk /dev/mapper/pve-root: 36.2 GB, 36238786560 bytes
255 heads, 63 sectors/track, 4405 cylinders, total 70778880 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/pve-root doesn't contain a valid partition table

Disk /dev/mapper/pve-swap: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders, total 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/pve-swap doesn't contain a valid partition table

Disk /dev/mapper/pve-data: 75.9 GB, 75937873920 bytes
255 heads, 63 sectors/track, 9232 cylinders, total 148316160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/pve-data doesn't contain a valid partition table

fdisk -l on pve2

Code:

root@pve2:~# fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 146.0 GB, 145999527936 bytes
255 heads, 63 sectors/track, 17750 cylinders, total 285155328 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1   285155327   142577663+  ee  GPT

Disk /dev/sdb: 1796.4 GB, 1796370071552 bytes
57 heads, 28 sectors/track, 2198330 cylinders, total 3508535296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xa14806fa

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3508535295  1754266624   83  Linux

Disk /dev/mapper/pve-root: 36.2 GB, 36238786560 bytes
255 heads, 63 sectors/track, 4405 cylinders, total 70778880 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/pve-root doesn't contain a valid partition table

Disk /dev/mapper/pve-swap: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders, total 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/pve-swap doesn't contain a valid partition table

Disk /dev/mapper/pve-data: 75.9 GB, 75937873920 bytes
255 heads, 63 sectors/track, 9232 cylinders, total 148316160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/pve-data doesn't contain a valid partition table

/dev/sdb - Storage System.

At the first node performs the following:

Code:

root@pve1:~# pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created
root@pve1:~# vgcreate storage1 /dev/sdb1
  Clustered volume group "storage1" successfully created
root@pve1:~# pvs
  PV         VG       Fmt  Attr PSize   PFree
  /dev/sda3  pve      lvm2 a--  135.47g 16.00g
  /dev/sdb1  storage1 lvm2 a--    1.63t  1.63t
root@pve1:~# vgs
  VG       #PV #LV #SN Attr   VSize   VFree
  pve        1   3   0 wz--n- 135.47g 16.00g
  storage1   1   0   0 wz--nc   1.63t  1.63t

but on the second node pvs and vgs display nothing:

Code:

root@pve2:~# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda3  pve  lvm2 a--  135.47g 16.00g
root@pve2:~# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  pve    1   3   0 wz--n- 135.47g 16.00g

In Web panel add storage LVM, choose volume group storage1. First node connects and works well with storage.

Any idea why not see the second node storage?

Best Regards.

m.ardito · Sep 24, 2014

Hi, try posting your cat /etc/pve/storage.cfg

LVM is defined as shared (web gui shows that)?
The simplest setup is to connect to iSCSI volume and build local LVM on that

is your storage a DS3524 ? I can't find DH3524 on google...?

Marco

Kirill · Sep 24, 2014

Code:

root@pve1:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content images,iso,vztmpl,rootdir
        maxfiles 0

lvm: str1
        vgname storage1
        shared
        content images

Code:

root@pve2:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content images,iso,vztmpl,rootdir
        maxfiles 0

lvm: str1
        vgname storage1
        shared
        content images

IBM DS3524

I tried shared, and no shared in web GUI. But now is shared.

m.ardito · Sep 24, 2014

I guess you know of this page
http://pve.proxmox.com/wiki/Storage_Model#Storage_types

there is
Network storage types supported

LVM Group (network backing with iSCSI targets)

and

Local storage types supported

LVM Group (local backing devices like block devices, FC devices, DRBD, etc.)

so it could be that without iSCSI behind, LVM is a "local" storage, and your DS3524 can't be shared.

I've always used LVM+iSCSI, my setup is like

iscsi: vm_ts879
target iqn.2004-04.com.qnap:ts-879u-rp:iscsi.pve.d4e6fc
portal ts879
content none

lvm: vm_disks_ts879
vgname pve_vm_disks_ts879
base vm_ts879:0.0.0.scsi-36001654c0d33887de342d4f29d88b1d3
shared
content images

Marco

Kirill · Sep 24, 2014

My variant is:
Local storage types supported

LVM Group (local backing devices like block devices, FC devices, DRBD, etc.)

But node pve2 don't seen storage.
Web gui in attached.

Kirill · Oct 1, 2014

I forgot install multipath-tool.

Now work fine.

Search

Search

Cluster and IBM storage

Kirill

New Member

m.ardito

Famous Member

Kirill

New Member

m.ardito

Famous Member

Kirill

New Member

Attachments

Kirill

New Member

We value your privacy