Trouble using glusterfs with libgfapi

[nountevo]

Greetings,

I have a working 3 nodes proxmox cluster.
Using glusterfs as storage without problem (created a manual mount point + a shared directory.)
Proxmox and glusterfs share the same nodes.

Now i would like to use libgfapi and get rid of fuse overhead, but it can't get it to work..
I added a storage type "glusterfs" in proxmox ui, but i always get a timeout while trying to create a vm.

TASK ERROR: create failed - unable to create image: got lock timeout - aborting command

It seems the qemu process will stay there forever :

root 42228 1 0 11:59 ? 00:00:00 /usr/bin/qemu-img create -f raw gluster://proxmoxt-rrgfs/glusterstorage/images/117/vm-117-disk-1.raw 2097152K

I had the same problem when using the command qemu-img info :

root@proxmoxt1:~# qemu-img info gluster://proxmoxt-rrgfs:24007/glusterstorage/images/101/vm-101-disk-1.raw 2>/dev/null
image: gluster://proxmoxt-rrgfs:24007/glusterstorage/images/101/vm-101-disk-1.raw
file format: raw
virtual size: 56G (60129542144 bytes)
disk size: 6.0G

It works fine when i add port 24007, but process hangs without it.
I tried a strace on glusterd process, and it won't receive anything when the port is not explicitly forced to 24007.

Is this the normal behaviour ? Is there a workaround to force the port used by proxmox ?


root@proxmoxt1:~# dpkg -l| grep gluster
ii glusterfs-client 3.5.1-1 amd64 clustered file-system (client package)
ii glusterfs-common 3.5.1-1 amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.5.1-1 amd64 clustered file-system (server package)

root@proxmoxt1:~# pveversion -v
proxmox-ve-2.6.32: not correctly installed (running kernel: 3.10.0-3-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-3.10.0-3-pve: 3.10.0-11
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-16
pve-firmware: 1.1-3
libpve-common-perl: 3.0-18
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.1-1

Regards,
Alexandre.

 

[spirit]

Hi,
Do you have tried with gluster 3.4.1, provided by proxmox repository ?

 

[nountevo]

Hello,

I did not try glusterfs 3.4.2 with libgfapi.
I wanted 3.5 for "Brick failure detection" and because i use one brick per disk (no raid), but it seems that release is not that good..

I guess there is a bug with qemu/libgfapi in 3.5 but i can't find it anywhere..
By the way, someone else is having the same issue here : https://botbot.me/freenode/gluster/2014-07-16/?msg=18137881&page=2

Regards,
Alexandre.

 

[brad_mssw]

You'll find a couple of other forum posts here about the same thing. 3.5 doesn't work, 3.4 does. However, I gave up on gluster as my VMs would report disk i/o errors on reboot of nodes randomly. I think it was due to disabling client-side quorum, but leaving server side quorum enabled. With only 3 nodes, you'd have to use replica 3 to be able to enable client-side quorum due to some sub-volume quorum check and that seemed wasteful. I've been happier with Ceph.

 

[nountevo]

With 3.5.0, glusterfsd processes crash randomly under load..
Well, 3.5.1 fixed a number of this bugs.

I had no problems with reboots, but indeed use 3 nodes with replica 3.
As seen on the official site "When client side quorum is enabled, there exists the rule such that atleast half of the bricks in the replica group should be UP and running. If not, the replica group would become read-only"

 

[nountevo]

I remember why i ended installing glusterfs 3.5...I believe 3.4 didn't support "backup-volfile-servers" (with an "s") mount option.

For now i reverted to 3.4.2 and it seems to work with libgfapi.

root@proxmoxt1:~# qemu-img info gluster://proxmoxt-rrgfs/glusterstorage/images/101/vm-101-disk-1.raw
image: gluster://proxmoxt-rrgfs/glusterstorage/images/101/vm-101-disk-1.raw
file format: raw
virtual size: 56G (60129542144 bytes)
disk size: 3.5G