Setting up GlusterFS on ProxMox hardware nodes...

[oeginc]

I've read that the latest version of ProxMox supports GlusterFS and I'd like to setup my /backup partition (to start with, and possible my /var/lib/vz partition if it works well) as GlusterFS filesystems.

I have 10 nodes, each node has a 2TB drive mounted on /backup. I'd like to make them all part of a RAID-10 like setup under GlusterFS so I have redundancy and hopefully some increased speed to help eliminate some bottlenecks.

That being said, I am running the latest release of ProxMox

# pveversion -v​

proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)​

pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)​

pve-kernel-2.6.32-26-pve: 2.6.32-114​

lvm2: 2.02.98-pve4​

clvm: 2.02.98-pve4​

corosync-pve: 1.4.5-1​

openais-pve: 1.1.4-3​

libqb0: 0.11.1-2​

redhat-cluster-pve: 3.2.0-2​

resource-agents-pve: 3.9.2-4​

fence-agents-pve: 4.0.0-2​

pve-cluster: 3.0-8​

qemu-server: 3.1-8​

pve-firmware: 1.0-23​

libpve-common-perl: 3.0-8​

libpve-access-control: 3.0-7​

libpve-storage-perl: 3.0-17​

pve-libspice-server1: 0.12.4-2​

vncterm: 1.1-4​

vzctl: 4.0-1pve4​

vzprocps: 2.0.11-2​

vzquota: 3.1-2​

pve-qemu-kvm: 1.4-17​

ksm-control-daemon: 1.1-1​

glusterfs-client: 3.4.1-1​

But when I try to install: glusterfs-server I get this strange error message (note glusterfs-client v3.4 is already installed)

# apt-get update; apt-get upgrade -y; apt-get install glusterfs-server​

Reading package lists... Done​

Building dependency tree​

Reading state information... Done​

Some packages could not be installed. This may mean that you have​

requested an impossible situation or if you are using the unstable​

distribution that some required packages have not yet been created​

or been moved out of Incoming.​

The following information may help to resolve the situation:​

​

The following packages have unmet dependencies:​

glusterfs-server : Depends: glusterfs-common (>= 3.2.7-3+deb7u1) but it is not going to be installed​

Depends: glusterfs-client (>= 3.2.7-3+deb7u1) but it is not going to be installed​

E: Unable to correct problems, you have held broken packages.​

You can see it says it depends on 'glusterfs-client' with a version greater than or equal to 3.2.7-3+deb7u1

On top of that, if it says it can't install either of them because the packages don't exist, but if you check

# apt-cache show glusterfs-client
​

Package: glusterfs-client​

Source: glusterfs​

Version: 3.2.7-3+deb7u1
...​

You can clearly see it's available in the repository... I'm not really sure what's going on here..

Thanks!

 

[oeginc]

I have found the latest debian packages, but they are 3.4.2-1, I can't find the version ProxMox is using. Do you think it would be a problem just dpkg -i installing the latest common, client and server?

 

[tom]

we use currently 3.4.1
(e.g. see http://download.proxmox.com/debian/dists/wheezy/pve-no-subscription/binary-amd64/)

3.4.2 is not yet tested here.

 

[oeginc]

Yeah, I just found that download site just before you posted this reply. Thanks! I didn't really want to get into changing out ProxMox packages...

 

[SamTzu]

I'm also running in to the same problem. Did you find a solution?

 

[tom]

Just to note, pvetest repo contains now latest 3.4.2 packages now (client and server)!

dpkg -l gluster*

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name             Version       Architecture  Description
+++-================-=============-=============-=====================================
ii  glusterfs-client 3.4.2-1       amd64         clustered file-system (client package
ii  glusterfs-common 3.4.2-1       amd64         GlusterFS common libraries and transl
ii  glusterfs-server 3.4.2-1       amd64         clustered file-system (server package

 

[SamTzu]

I found the glusterfs-server from here and after installing it manually I got server2server replication working.
wget http://download.proxmox.com/debian/...nary-amd64/glusterfs-server_3.4.0-2_amd64.deb

 

[oeginc]

I'm also running in to the same problem. Did you find a solution?

@SamTzu - Yes, the download link was posted above, but here it is in case you missed it.

http://download.proxmox.com/debian/...nary-amd64/glusterfs-server_3.4.1-1_amd64.deb

Just do a wget on that and install...

# wget http://download.proxmox.com/debian/...nary-amd64/glusterfs-server_3.4.1-1_amd64.deb
# dpkg -i glusterfs-server_3.4.1-1_amd64.deb

Then configure and run Gluster as normal. I'm not sure if it's the "right" way to do it, but what I did is manually configured gluster outside of ProxMox on each node to mount my Gluster Filesystem on /backup, then inside ProxMox I just told it that /backup was a local directory...

I'd like to migrate the /var/lib/vz directory next, but I've heard some people have had trouble running containers and/or VM's from a GlusterFS partition, anyone have experience here? I'd imagine there *might* be a problem when you are trying to move a container/vm to a new host, the migration process might get confused and think the machine is already on the destination server (or is that was the "shared" flag is for in the storage options? and does vzmigrate honor that?)

Anyways.. To start I ended up using 6 of the servers in a stripe 3 / replica 2 configuration and it worked well. I ended up with ~5.2 TB of hard drive space available (out of 12TB to start with) and was getting about 70MB/s - 80MB/s write speeds, which wasn't too bad. I'm going to play around with the configuration a bit to try to find a comfortable median between speed and losing 50% of my drive space...

 

[tom]

@SamTzu - Yes, the download link was posted above, but here it is in case you missed it.

http://download.proxmox.com/debian/...nary-amd64/glusterfs-server_3.4.1-1_amd64.deb

Just do a wget on that and install...

# wget http://download.proxmox.com/debian/...nary-amd64/glusterfs-server_3.4.1-1_amd64.deb
# dpkg -i glusterfs-server_3.4.1-1_amd64.deb

...

not sure if I miss something here but why don´t you just:

> apt-get install glusterfs-server

if you add pvetest in sources.list, you will get latest packages (3.4.2)

 

[oeginc]

not sure if I miss something here but why don´t you just:

> apt-get install glusterfs-server

if you add pvetest in sources.list, you will get latest packages (3.4.2)

Only because I'm trying to keep my ProxMox install pure.

 

[tom]

Only because I'm trying to keep my ProxMox install pure.

I just want to tell that you do NOT need to manually download gluster packages with wget and install with dpkg -i

just use apt-get install PACKAGE, whatever repository you like. the stable repos contains gluster 3.4.1, pvetest latest 3.4.2.

 

[barryflanagan]

I just want to tell that you do NOT need to manually download gluster packages with wget and install with dpkg -i

just use apt-get install PACKAGE, whatever repository you like. the stable repos contains gluster 3.4.1, pvetest latest 3.4.2.

Actually, no. The pve-no-subscription repository (not enabled by default) does contain the 3.4.1 glusterfs-server but unless you have manually enabled that repo the only server you are offered is 3.2.7 (even though the client is 3.4.1.) and it will not install due to dependency issues.

Regards,

-Barry Flanagan

 

[tom]

Actually, no. The pve-no-subscription repository (not enabled by default) does contain the 3.4.1 glusterfs-server but unless you have manually enabled that repo the only server you are offered is 3.2.7 (even though the client is 3.4.1.) and it will not install due to dependency issues.

Regards,

-Barry Flanagan

the enterprise subscription is enabled by default, working as soon as you upload your key.

if you do not want to use the subscription services, you need to configure the pve-no-subscription IN ANY CASE, not only if you want to use gluster.

so configure your update repo as needed, then your will get the packages. without repo, no packages can be delivered - hope its clear now.

 

[barryflanagan]

the enterprise subscription is enabled by default, working as soon as you upload your key.

if you do not want to use the subscription services, you need to configure the pve-no-subscription IN ANY CASE, not only if you want to use gluster.

so configure your update repo as needed, then your will get the packages. without repo, no packages can be delivered - hope its clear now.

Ah, OK. I did not realize that. Possibly I missed it because as far as I can see this is not mentioned in the installation instructions, and the Wiki (http://pve.proxmox.com/wiki/Package_repositories) states:

"The pve-no-subscription repo can be used for testing and non-production use. Its not recommended to run on production servers as these packages are not always heavily tested and validated."

Thanks for the clarification.

-Barry

 

[vcp_ai]

Then configure and run Gluster as normal. I'm not sure if it's the "right" way to do it, but what I did is manually configured gluster outside of ProxMox on each node to mount my Gluster Filesystem on /backup, then inside ProxMox I just told it that /backup was a local directory...

If you mount gluster share as local, you will miss the feature of Proxmox to been able to use directly a Gluster File System. I.E. it will be more or less as if you mount an NFS share , and Proxmox implementation of Gluster is (should be) more efficient !!!

This is what I understand, please correct me if I'm wrong !!

Mi plan (when I have time and a little cash) for my test-lab is to install glusterfs-server in 2 proxmox serves (gluster RAID1), and use it from this 2 and 2 more (without disk) servers , but allways as a glusterfs share.