Online migrations fails for CT on GlusterFS

[GomoX]

Hello,

I set up shared storage on my Proxmox cluster using GlusterFS and everything appears to work fine with VMs. I can perform both live and offline migration between nodes stored there.

Now, for containers things don't appear to work as well. When trying to live migrate I get the following error:

Dec 06 10:52:01 starting migration of CT 128 to node 'srv-counter' (10.10.99.30)
Dec 06 10:52:01 container is running - using online migration
Dec 06 10:52:01 container data is on shared storage 'gluster'
Dec 06 10:52:01 start live migration - suspending container
Dec 06 10:52:11 # vzctl --skiplock chkpnt 128 --suspend
Dec 06 10:52:01 Setting up checkpoint...
Dec 06 10:52:01 suspend...
Dec 06 10:52:11 Can not suspend container: Interrupted system call
Dec 06 10:52:11 Error: interrupted or timed out.
Dec 06 10:52:11 ERROR: Failed to suspend container: Checkpointing failed
Dec 06 10:52:11 aborting phase 1 - cleanup resources
Dec 06 10:52:11 start final cleanup
Dec 06 10:52:11 ERROR: migration aborted (duration 00:00:11): Failed to suspend container: Checkpointing failed
TASK ERROR: migration aborted

Keep in mind that live migration through local storage works just fine for CTs.

I am running Proxmox 2.2 and GlusterFS 3.3 (latest packages of everything from the official repos).

pve-manager: 2.2-31 (pve-manager/2.2/e94e95e9)
running kernel: 2.6.32-16-pve
proxmox-ve-2.6.32: 2.2-82
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-12-pve: 2.6.32-68
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-33
qemu-server: 2.0-69
pve-firmware: 1.0-21
libpve-common-perl: 1.0-39
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.2-7
ksm-control-daemon: 1.1-1

Any hints on what might be causing this?

 

[GomoX]

GLHMarmot on IRC is telling me that he has this running fine on pvetest. I'd like to avoid that if possible, so any hints would be appreciated.

 

[GomoX]

Any ideas on this?

 

[andydills]

I have this same problem. Would love to get an answer, this is kind of frustrating. I'm tempted to just make everything a KVM, but it would be nice to have this working.

 

[tom]

GlusterFS is not that heavily used with Proxmox VE, for containers we recommend local storage (ext3) or NFS. We at Proxmox does not test GlusterFS in our test labs.

 

[andydills]

Is there a "best solution" for drbd volumes paired with containers? Long term I would be migrating to shared storage, but for now drbd makes more sense.

 

[tom]

no, DRBD on Proxmox VE is always with some limitations. If you know exactly what you do, DRBD could be great way. But this thread is not about DRBD - its about online migration with glusterFS.

so if you have new questions, pls open a new thread.

 

[GomoX]

GlusterFS is not that heavily used with Proxmox VE, for containers we recommend local storage (ext3) or NFS. We at Proxmox does not test GlusterFS in our test labs.

Hey tom, is there a reason for this? Native Gluster is a great fit for a Proxmox cluster without all the pain in setting up virtual IP's to serve a Gluster volume over HA NFS (there are quite a few pitfalls involved). Is there anything that can be done to "push" Gluster to be a more supported option?

 

[tom]

Hey tom, is there a reason for this? Native Gluster is a great fit for a Proxmox cluster without all the pain in setting up virtual IP's to serve a Gluster volume over HA NFS (there are quite a few pitfalls involved). Is there anything that can be done to "push" Gluster to be a more supported option?

If you want to encourage others to use GlusterFS, I suggest you create a wiki page, describing the whole setup and usage (If you start this, one of our team will review the wiki page)

 

[GomoX]

If you want to encourage others to use GlusterFS, I suggest you create a wiki page, describing the whole setup and usage (If you start this, one of our team will review the wiki page)

Sounds good, will do.

 

[vcp_ai]

Also interested...
Waiting your work.