Proxmox, GlusterFS 2 and FUSE

[Myatu]

(Note: this is resolved. Solution at the end.)

Okay, so I'm playing with an experimental setup where one VM is a GlusterFS server and another VM a GlusterFS client (the idea is to replace the current NFS with GlusterFS).

It's *almost* working, I'm just running into a problem with FUSE on the "client" VM, which is actually more important than the server part... Here's what I have done so far (would make a nice Wiki if this finally works):

To prepare the GlusterFS Server VM ("Server", VEID 105) I did this on the Host Node:

# vzctl set 105 --capability sys_admin:on --save
# vzctl restart 105

(Note: without "sys_adminn" I get errors starting the GlusterFS daemon.)

Then I enter the Server (debian-5.0-standard_5.0-1 template) and do the following:

# aptitude install build-essential flex bison libdb4.6-dev libibverbs-dev
# wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-2.0.0rc7.tar.gz
# tar -xzf glusterfs-2.0.0rc7.tar.gz
# cd glusterfs-2.0.0rc7 
# ./configure --prefix=/usr --disable-fuse-client >/dev/null
# make && make install

I used the "--disable-fuse-client" option, as I do not need it for a dedicated GlusterFS server. Additionally, the "libibverbs-dev" package is not required if you're not using Infiniband.

I create a simple configuration:

# cat /etc/glusterfs/glusterfs-server.vol
volume brick
  type storage/posix
  option directory /data/test
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.brick.allow * # No restrictions
  subvolumes brick
end-volume

Make the directory:

# mkdir -p /data/test

Finally fire up the GlusterFS Daemon on the server:

# glusterfs -f /etc/glusterfs/glusterfs-server.vol -l /var/log/glusterfs.log

Okay, so that's the Server squared away. On to the the GlusterFS Client.

To prepare the GusterFS Client VM ("Client", VEID 106), I did this on the Host Node:

# modprobe fuse
# lsmod | grep fuse
fuse                   65680  2
# vzctl set 106 --devices c:10:229:rw --save
# vzctl exec 106 mknod /dev/fuse c 10 229

I enter the Client (debian-4.0-i386-minimal template) and first get the required packages to compile:

# aptitude -y install build-essential flex bison libdb4.6-dev libibverbs-dev

Next up, I compile the GlusterFS-patched FUSE library:

# wget http://ftp.gluster.com/pub/gluster/glusterfs/fuse/fuse-2.7.4glfs11.tar.gz
# tar -xzf fuse-2.7.4glfs11.tar.gz
# cd fuse-2.7.4glfs11
# ./configure --prefix=/usr --disable-kernel-module
# make && make install

(Note: --disable-kernel-module is required, given that it's inside a VM container).

Then, I compile GlusterFS for the Client:

# wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-2.0.0rc7.tar.gz
# tar -xzf glusterfs-2.0.0rc7.tar.gz
# cd glusterfs-2.0.0rc7 
# ./configure --prefix=/usr >/dev/null
GlusterFS configure summary
===========================
FUSE client        : yes
Infiniband verbs   : yes
epoll IO multiplex : yes
Berkeley-DB        : yes
libglusterfsclient : yes
mod_glusterfs      : no ()
argp-standalone    : no
# make && make install

The configuration file looks as following:

# cat /etc/glusterfs/glusterfs-client.vol
volume test
  type protocol/client
  option transport-type tcp
  option remote-host 10.10.101.5 # IP address of the Server
  option remote-subvolume brick
end-volume

And finally, I create a mount point:

# mkdir /mnt/test

Now, at *this* point is where I'm hitting a wall. I should be able to do a mount like this:

# mount -t glusterfs /etc/glusterfs/glusterfs-client.vol /mnt/test

But no such luck. So I had a look at what's going on with:

# glusterfs -f /etc/glusterfs/glusterfs-client.vol -l /var/log/glusterfs.log -L DEBUG /mnt/test

And the log file will return this:

Version      : glusterfs 2.0.0rc7 built on Mar 30 2009 18:42:28
TLA Revision : 4e5c297d7c3480d0d3ab1c0c2a184c6a4fb801ef
Starting Time: 2009-03-30 18:47:26
Command line : glusterfs -f /etc/glusterfs/glusterfs-client.vol -l /var/log/glusterfs.log -L DEBUG /mnt/test
PID          : 21370
System name  : Linux
Nodename     : gfsclient
Kernel Release : 2.6.24-2-pve
Hardware Identifier: i686

Given volfile:
( ... cut out  ... )

2009-03-30 18:47:26 D [spec.y:188:new_section] parser: New node for 'test'
2009-03-30 18:47:26 D [xlator.c:469:xlator_set_type] xlator: attempt to load file /usr/lib/glusterfs/2.0.0rc7/xlator/protocol/client.so
2009-03-30 18:47:26 D [spec.y:214:section_type] parser: Type:test:protocol/client
2009-03-30 18:47:26 D [spec.y:243:section_option] parser: Option:test:transport-type:tcp
2009-03-30 18:47:26 D [spec.y:243:section_option] parser: Option:test:remote-host:10.10.101.6
2009-03-30 18:47:26 D [spec.y:243:section_option] parser: Option:test:remote-subvolume:brick
2009-03-30 18:47:26 D [spec.y:327:section_end] parser: end:test
2009-03-30 18:47:26 D [xlator.c:469:xlator_set_type] xlator: attempt to load file /usr/lib/glusterfs/2.0.0rc7/xlator/mount/fuse.so
2009-03-30 18:47:26 D [glusterfsd.c:1115:main] glusterfs: running in pid 21371
2009-03-30 18:47:26 D [client-protocol.c:6559:init] test: defaulting transport-timeout to 600
2009-03-30 18:47:26 D [client-protocol.c:6570:init] test: defaulting ping-timeout to 10
2009-03-30 18:47:26 D [transport.c:141:transport_load] transport: attempt to load file /usr/lib/glusterfs/2.0.0rc7/transport/socket.so
2009-03-30 18:47:26 D [transport.c:141:transport_load] transport: attempt to load file /usr/lib/glusterfs/2.0.0rc7/transport/socket.so
2009-03-30 18:47:26 D [xlator.c:599:xlator_init_rec] test: Initialization done
2009-03-30 18:47:26 E [fuse-bridge.c:2838:init] glusterfs-fuse: fuse_mount() failed with error No such device on mount point /mnt/test
2009-03-30 18:47:26 E [xlator.c:611:xlator_init_rec] xlator: initialization of volume 'fuse' failed, review your volfile again
2009-03-30 18:47:26 E [glusterfsd.c:494:_xlator_graph_init] glusterfs: initializing translator failed
2009-03-30 18:47:26 E [glusterfsd.c:1127:main] glusterfs: translator initialization failed.  exiting

As you can see, it fails at "No such device on mount point /mnt/test", yet I did the "vzctl set 106 --devices c:10:229:rw --save" followed by a "mknod /dev/fuse c 10 229" on the client. /dev/fuse shows up OK.

tune2fs reports the following error:

# tune2fs -l /dev/fuse
tune2fs 1.41.3 (12-Oct-2008)
tune2fs: Illegal seek while trying to open /dev/fuse
Couldn't find valid filesystem superblock.

... And I did the "modprobe fuse" on the HN.

... So, what's the last step I'm missing?

[RESOLVED] -Aha! I answered my own question here. The answer is that in addition to the server, a client with GlusterFS also needs "--sysadminn". Thus, on the Host Node (where 106 is the VEID of the Client) you use:

# vzctl set 106 --capability sys_admin:on --save
# vzctl restart 106

After this, everything works like a charm. GlusterFS is much faster than NFS (especially in a container) and it gives you a whole range new options! Have a look at http://www.gluster.org.

One thing though: what's the drawback of "sys_adminn" and is it possible to do this differently?

 

[dietmar]

One thing though: what's the drawback of "sys_adminn" and is it possible to do this differently?

Please can you ask on the openvz list (real openvz experts are there).

- Dietmar

 

[Myatu]

Good morning Dietmar,

Thanks for the great product, by the way.

I've looked at the discussions at OpenVZ and it's a bit of a mixed bag. Overall, it should not be a problem for the general user unless things are done unchecked. I don't see any major concerns.

There have been talks to make the sys_admin capability a little more fine grained as well, as currently it encompasses too many things which aren't neccesarily needed.

 

[dcalvache]

i have feedback about glusterFS have a lot of perfomance issues when managing big file shares. (aprox 150gb at least ), can you test this?

How big is the file share you have mount?


Greetings and WD howto

 

[unlocksmith]

In 2.0, if you have files larger than 1GB, io-cache is known to slow down and or cause higher CPU utilization. Just comment out io-cache translator, this problem will go away. In 3.0 io-cache translator uses rbtree mechanism and can cache very large files efficiently. 3.0 is in beta currently. Stable version will be available in December.