Proxmox, ceph & HP P2000 FC

[x0139]

Hello guys.
I setting up a new cluster with 4 blade servers.
For storage use a HP P2000 G3, connected via fibre channel to hp chassis.
I connect 2 vg's per server (8 disks total).
And now, when i starting to install the ceph, i have a problem - on each server my 2 vgs is quad - instead 2 disks i see 8.
What happen? How can i change this to view only my 2 disks on each server?

 

[x0139]

i found my disk on /dev/dm-5 and /dev/dm-6 devices.
but i cannot add them via pveceph createosd - have an error "unable to get device info for 'dm-5'"

i've tried do this by ceph osd create, but cannot start them (((

 

[udo]

Hello guys.
I setting up a new cluster with 4 blade servers.
For storage use a HP P2000 G3, connected via fibre channel to hp chassis.
I connect 2 vg's per server (8 disks total).
And now, when i starting to install the ceph, i have a problem - on each server my 2 vgs is quad - instead 2 disks i see 8.
What happen? How can i change this to view only my 2 disks on each server?

Hi,
don't know the P2000 G3 but looks at the specs...
Why you want this SAN-Storage use for Ceph?? Makes no realy sense for me (exept, there are other pve-nodes outside of the blade-center).
The P2000 support raided volumes to all Nodes, which you can use for lvm (thin).
Should be fast! With an ceph-layer between the storage will be much slower and not more failsafe, because the data are stored on the same device...

Udo

 

[x0139]

In our organization, we already have the P2000 storage.
Hmm... u say that lvm be much faster than ceph?
where i can read about it?

 

[mir]

In our organization, we already have the P2000 storage.
Hmm... u say that lvm be much faster than ceph?
where i can read about it?

Ceph performance is a derived function of the number of disks and storage servers. Performance wise to compete against a SAN will require a lot of servers and a lot of disks.

 

[x0139]

Thanks, uunderstood.
do u have a link to manual how to configure proxmox with lvm thin storage?
official proxmox wiki does not work ((

 

[mir]

Thanks, uunderstood.
do u have a link to manual how to configure proxmox with lvm thin storage?
official proxmox wiki does not work ((

Proxmox will (from the gui), unless otherwise stated, make a thin provisioned lvm.

 

[x0139]

Can not find the right manual howto setup lvm on shared storage (via FC).
Can anyone give some links?

 

[udo]

Can not find the right manual howto setup lvm on shared storage (via FC).
Can anyone give some links?

Hi,
it's the same like with local storage - pve will handle the locking of the LVs...
Simply create an VG on the FC-Disk (command line), after that you can create lvm-thin on this and use this in der Datacenter->Storage definition.

Udo

 

[x0139]

Hi,
it's the same like with local storage - pve will handle the locking of the LVs...
Simply create an VG on the FC-Disk (command line), after that you can create lvm-thin on this and use this in der Datacenter->Storage definition.

Udo

Greetings, udo!
So, i have a RAID 5 disk in HP P2000.
I map it on all my 4 blades via hp storage console, and then just work with them like the local storage - create on one server lvm-thin group and add this storage from proxmox web-console?

 

[udo]

Greetings, udo!
So, i have a RAID 5 disk in HP P2000.
I map it on all my 4 blades via hp storage console, and then just work with them like the local storage - create on one server lvm-thin group and add this storage from proxmox web-console?

Hi,
yes. FC-Storage, which is connected to more than one node, is shared storage.
In the wiki (storage-modell) is FC unfortunally as local storage only discribed...
Here it's described with iscsi https://pve.proxmox.com/wiki/Storage_Model#LVM_Groups_-_An_Ideal_Solution but with FC it's the same without Step 1 (iscsi target).
Its the same as with an local disk - except, that the storage is marked as shared.

I would create lvm-thin on the FC-VG.

Udo

 

[x0139]

Hi,
yes. FC-Storage, which is connected to more than one node, is shared storage.
In the wiki (storage-modell) is FC unfortunally as local storage only discribed...
Here it's described with iscsi https://pve.proxmox.com/wiki/Storage_Model#LVM_Groups_-_An_Ideal_Solution but with FC it's the same without Step 1 (iscsi target).
Its the same as with an local disk - except, that the storage is marked as shared.

I would create lvm-thin on the FC-VG.

Udo

Big thx, udo!
I ve created a LVM - its work fine, shared storage is great - online migration working.
But, when i've tried to use LVM-thin - and i cannot find the "Shared" check-button. and online migration is not working for me ((
So i will uuse LVM, its all ok.

Buut now i have another problem - Fencing.
In all manuals (in official too) wrote:

• Enable fencing in /etc/default/redhat-cluster-pve (Just uncomment the last line, see below):

But i have not this file /etc/default/redhat-cluster-pve.

May be its need to install some fencing tools?

============================================
added some time later...

I've juust tried to reboot host node with VM in HA-mode - and migrating and staring vm's on dead node is work.

not sure about installing fencing by official manual...
think its work from out of the box in Proxmox 4.2

 

[udo]

Big thx, udo!
I ve created a LVM - its work fine, shared storage is great - online migration working.
But, when i've tried to use LVM-thin - and i cannot find the "Shared" check-button. and online migration is not working for me ((
So i will uuse LVM, its all ok.

Buut now i have another problem - Fencing.
In all manuals (in official too) wrote:

• Enable fencing in /etc/default/redhat-cluster-pve (Just uncomment the last line, see below):

But i have not this file /etc/default/redhat-cluster-pve.

May be its need to install some fencing tools?

============================================
added some time later...

I've juust tried to reboot host node with VM in HA-mode - and migrating and staring vm's on dead node is work.

not sure about installing fencing by official manual...
think its work from out of the box in Proxmox 4.2

Hi,
about lvm-thin:
you have some advantages with lvm-thin, like snapshot (very nice feature).
Normal lvm is also ok (I use'd this for a long long time) but I missed snapshot some times.

Fencing:
I don't have experiences with fencing... perhaps you should open an new thread for that.
But first use the search-function for fencing and ha - I think you will find a lot?!

Udo