File system type on shared storage (SAN IBM DS3512)

sv63rus

New Member
Jun 18, 2012
21
0
1
Russia
Hello!!

I have two servers IBM x3650 connected to IBM System Storage DS3512 with HBA SAS adapter. I config multipath-tools and each server have access to same shared LUN on storage.
On this LUN i mkfs ext3 and mounted on each server in /mnt/vol1. In proxmox web-management interface i add /mnt/vol1 to storage by "Directory" and check "shared". VM's and OpenVZ place on /mnt/vol1

And now i have problem with migrate openVZ conteiners... i think because have no dynamic update file system. When i make dir on on node second node don't see this dir until /mnt/vol1 not remounted.

Pease help! What type of file system to choose to make it work?
 
Last edited:
Thanks for the quick response! Let me ask a few more questions on the shared drive?
sorry for my english.

This means that the live migration openvz impossible? If possible, how?

what options there are connections to the shared LUN on storage system for live migration?
qemu can live migrate the VM if the storage added using LVM groups and each virtual machine has its own lv ?
Is it possible to use ext3 or ext4 on top of shared lvm ... or clvm ?
Maybe I can use gfs2? I installed gfs2-tools and was able mkfs -t gfs2 ... / dev/mapper/mpath0, but I can not mount with errors

I like Proxmox, and I would like to use it in production.

I have 2 OpenVZ (centos) and 1 qemu vm (win 2003).
fenceng device also have. I need create HA cluster.

Thanks for the help!
Best regards,
Valerie
 
This means that the live migration openvz impossible? If possible, how?
Yes impossible in the sense of uninterrupted service. A VZ container will be suspended for a number of seconds (The time to transfer the state)
what options there are connections to the shared LUN on storage system for live migration?
Using KVM for VM and you are find and dandy. Make a LVM device un-top of the LUN and mark this LVM device shared.
Is it possible to use ext3 or ext4 on top of shared lvm ... or clvm ?
Maybe I can use gfs2? I installed gfs2-tools and was able mkfs -t gfs2 ... / dev/mapper/mpath0, but I can not mount with errors
ext3 and ext4 are both supported, ext3 is the recommended. gfs2 is unsupported but should work.
 
Yes impossible in the sense of uninterrupted service. A VZ container will be suspended for a number of seconds (The time to transfer the state)
what kind of storage type: iSCSI,NFS, local shared direcroty, LVM group and
where should I store the VZ containers for have the support of "live" migration? or offline migration ?
Let me remind you that I have a connection to the SAN with 2 SAS HBA adapters on each server and my server see LUN as the two devices are aggregated in failover multipath device /dev/mapper/mpath0



This weekend I will take are two tests for ext3 on top of lvm un-top of the LUN and gfs2 and write the results to this thread.

Thank's for help
 
Last edited:
...
This weekend I will take are two tests for ext3 on top of lvm un-top of the LUN and gfs2 and write the results to this thread.

Thank's for help
Hi,
to mount an ext-fs (of xfs...) simultanous on different nodes is very dangerous! Don't do that if you don't need data corruption.
For this thing you need an cluster-filesystem.

Other possibility for kvm is lvm-storage - the lv is only open by one node at one time. Easy and work well.

For OpenVz you can also use NFS, but you need an nfs-server and normaly you have than an spof (singe point of failure). Except you have an ha-nfs-server which is normaly oversized...

Udo
 
Last edited:
what kind of storage type: iSCSI,NFS, local shared direcroty, LVM group and
where should I store the VZ containers for have the support of "live" migration? or offline migration ?
Let me remind you that I have a connection to the SAN with 2 SAS HBA adapters on each server and my server see LUN as the two devices are aggregated in failover multipath device /dev/mapper/mpath0



This weekend I will take are two tests for ext3 on top of lvm un-top of the LUN and gfs2 and write the results to this thread.

Thank's for help
For KVM I would stick to Multipath LUN's. For VZ your only option is NFS if you want to be able to do online migration.
 
Except you have an ha-nfs-server which is normaly oversized...
ha-nfs-server sucks. File locks resides in kernel mode which means any active locks will hang the clients since your only options, if you want to avoid data loss, is hard mounts. Hard mounts can only be regained by rebooting the clients.

From man nfs:
"soft / hard Determines the recovery behavior of the NFS client after an NFS request times out. If neither option is speci-
fied (or if the hard option is specified), NFS requests
are retried indefinitely. If the soft option is speci-
fied, then the NFS client fails an NFS request after
retrans retransmissions have been sent, causing the NFS
client to return an error to the calling application.


NB: A so-called "soft" timeout can cause silent data
corruption in certain cases. As such, use the soft
option only when client responsiveness is more important
than data integrity. Using NFS over TCP or increasing
the value of the retrans option may mitigate some of the
risks of using the soft option."
 
For KVM I would stick to Multipath LUN's. For VZ your only option is NFS if you want to be able to do online migration.

And only for offline OpenVZ migration what is configuration fs and storage ?

I already have a SAN IBM DS3512 with shared LUN via SAS HBA and I want to use it as shared storages for vm and vz cont.
 
Last edited:
CT can use be stored on local storage or on NFS.
Migrations works on local storage, online and offline.

On NFS also (but currently there is an open bug here).
 
Thank you for response !
I understand that CT can use be stored on local storage or on NFS. but I buy SAN for get high perfomance.

If I use a local store that I can not have sufficient capacity and availability. If one node fail i can loss CT that stored on local storage.
 
Last edited:
Thank you for response !
I understand that CT can use be stored on local storage or on NFS. but I buy SAN for get high pefomance.

If I use a local store that I can not have sufficient capacity and availability. If one node fail i can loss CT that stored on local storage.
If you want to use your SAN and CT your only option currently is NFS. Work is in progress of other solutions but I sincerely doubt we will see these options in the near future, in stable I will not be surprised if we first see this in pve-3.0 since the underlying technologies is a moving target.
 
Please help!

I want make gfs2 on my shared SAN.

root@PVE-N2:~# pveversion -v
pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-66
pve-kernel-2.6.32-11-pve: 2.6.32-66
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-15
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1

root@PVE-N2:~# cman_tool
cman_tool: no operation specified
root@PVE-N2:~# cman_tool status
Version: 6.2.0
Config Version: 3
Cluster Name: XYZ
Cluster Id: 549
Cluster Member: Yes
Cluster Generation: 164
Membership state: Cluster-Member
Nodes: 2
Expected votes: 3
Total votes: 2
Node votes: 1
Quorum: 2
Active subsystems: 6
Flags:
Ports Bound: 0 11
Node name: PVE-N2
Node ID: 2
Multicast addresses: 239.192.2.39
Node addresses: 192.168.44.6

clvmd started, fence_tool join, /etc/default/redhat-cluster-pve FENCE_JOIN "yes", lvmconf --enable-cluster
/dev/mapper/mpath0 allow in filter in lvm.conf

i do pvcreate first on one node. pvcreate /dev/mapper/mapth0 successfully.
after this i do vgcreate store1_vg /dev/mapper/mpath0 all fine
but then i do lvcreate -l 100%FREE -n lvstore store1_vg i have error that vg store1_vg not found.

root@PVE-N1:~#vgremove store1_vg also bad -> store1_vg not found

root@PVE-N1:~#pvremove /dev/mapper/mpath0 --> PV /dev/mapper/mpath0 belongs to Volume Group store1_vg so please use vgreduce first.

after this i shutdown first node and try create pv vg lv on second node.

root@PVE-N2:~# pvcreate /dev/mapper/mpath0
Can't initialize physical volume "/dev/mapper/mpath0" of volume group "store1_vg" without -ff

root@PVE-N2:~# pvcreate -ff /dev/mapper/mpath0
Really INITIALIZE physical volume "/dev/mapper/mpath0" of volume group "store1_vg" [y/n]? y
WARNING: Forcing physical volume creation on /dev/mapper/mpath0 of volume group "store1_vg"
Writing physical volume data to disk "/dev/mapper/mpath0"
Physical volume "/dev/mapper/mpath0" successfully created

ok!

i try to create new vg

root@PVE-N2:~# vgcreate -c y -v storevg /dev/mapper/mpath0
Adding physical volume '/dev/mapper/mpath0' to volume group 'storevg'
Archiving volume group "storevg" metadata (seqno 0).
Creating volume group backup "/etc/lvm/backup/storevg" (seqno 1).
Clustered volume group "storevg" successfully created

I think thats ok ! but ...


root@PVE-N2:~# vgscan
Reading all physical volumes. This may take a while...

root@PVE-N2:~# vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 277.96g 16.00g

Where my new vg storevg ?? o_O

try create lv
root@PVE-N2:~# lvcreate -l 100%FREE -n storelv storevg
Volume group "storevg" not found

root@PVE-N2:~# pvremove /dev/mapper/mpath0
PV /dev/mapper/mpath0 belongs to Volume Group storevg so please use vgreduce first.
(If you are certain you need pvremove, then confirm by using --force twice.)

root@PVE-N2:~# vgreduce -f storevg /dev/mapper/mpath0
Volume group "storevg" not found
cluster request failed: Invalid argument
Internal error: Attempt to unlock unlocked VG storevg.


root@PVE-N2:~# pvcreate /dev/mapper/mpath0
Can't initialize physical volume "/dev/mapper/mpath0" of volume group "storevg" without -ff
 
Please help!

...
i try to create new vg

root@PVE-N2:~# vgcreate -c y -v storevg /dev/mapper/mpath0
Adding physical volume '/dev/mapper/mpath0' to volume group 'storevg'
Archiving volume group "storevg" metadata (seqno 0).
Creating volume group backup "/etc/lvm/backup/storevg" (seqno 1).
Clustered volume group "storevg" successfully created

...
Hi,
take a look at /etc/lvm/lvm.conf - I guess, that /dev/mpath0?? is not a device, where lvm looks for.

Udo
 
Hi Thank's ! I actually was understood.
but can understand why before in lvm filter i write /dev/mapper/mpath0 but now pv i can see as /dev/disk/by-id/dm-name-mpath0

And I can not decide what to write in fstab:
/dev/store/storelvm /mnt/vol1 gfs2 noatime,nodiratime 0 2
is that line in fstab mount point does not mount at startup, but manual after boot mount

root@pve-n1:~# ls /dev/mapper
control mpath0 pve-data pve-root pve-swap store-storelvm

Maybe I should write
/dev/mapper/store-storelvm /mnt/vol1 gfs2 noatime,nodiratime 0 2

or
root@pve-n1:~# pvs
PV VG Fmt Attr PSize PFree
/dev/disk/by-id/dm-name-mpath0 store lvm2 a-- 836.69g 0
/dev/sda2 pve lvm2 a-- 277.96g 16.00g

/dev/disk/by-id/dm-name-mpath0 /mnt/vol1 gfs2 noatime,nodiratime 0 2

please help !
 
Last edited:
Hi!
GFS2 earned but with a bunch of errors. Node does not turn off properly, hangs on unmounting. Boots normally only if the other node is running. I think that the GFS will work in a cluster of three nodes, but I have just two. In the end I decided not to use the GFS2 and use LVM as advised developers. It is a pity that you can not enjoy the performance openvz on shared storage. :(
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!