Proxmox 3.1 GlusterFS Setup, Confused by the process

[Freeheart]

What am I trying to do?
I'd like to have a GlusterFS filesystem replicate data across 3 partitions on SSD, each partition residing on a different node running Proxmox 3.1. I would like to expose this filesystem to all nodes via network as my ISO file store.

Background
I've installed Debian 7.1 on three servers, and followed the Proxmox 3.1 installation process without issues. These systems are all showing up in the Proxmox Web Admin panel, and are responding to commands properly. There are no VMs on any of the nodes.

Each system uses uEFI and GPT on a 120GB SSD as the first drive.
~100 MB EFI Boot partition
500 MB /boot on ext4
40 GB LVM volume - assigned to VG "system"
- 10 GB swap allocated as LV "swap"
- 30 GB root filesystem on ext4 allocated as LV "root"
40 GB LVM volume - assigned to VG "data"
- 40 GB ext4 filesystem as LV "ISO"

Additionally, each system has platter drives, but they're currently unformatted, unpartitioned, and unmounted (I'll save that for later!)

Where am I confused?
Other than simple testing of Gluster, and much more experience with Proxmox, I've never tried to integrate them.

1.) Should be creating a GlusterFS volume, peering it to nodes, and then plugging details of it into Proxmox?
2.) Should I create the storage in Proxmox, letting it handle everything including the creation of the GlusterFS peers? If this, how do I specify at creation to use a non-standard path?

Thanks in advance.

 

[spirit]

Hi,

you need to create the glusterfs manually and create a gluster volume.

Then you can configure it in proxmox.

 

[Freeheart]

Thanks.

I see where I went wrong now, and it's incredibly stupid.

I successfully created the GlusterFS volume, but it wasn't usable by Proxmox.

My stupid oversight was that even though I created, I never MOUNTED it locally.

Added an entry in /etc/fstab for a local mount (used /mnt/pve/iso, why not...) on each server, rebooted, and now it's usable space!

 

[vcp_ai]

Re: Proxmox 3.1 GlusterFS Setup, On SAME Servers as Promox

Hi,

you need to create the glusterfs manually and create a gluster volume.

Then you can configure it in proxmox.

Hi, I've created a test system for Glusterfs with a basic config:

2 KVM virtual machines (fedora19 minimal + Glusterfs-server)
Created a replica 2 volume and mounted it from (another) Proxmox.
Created a virtual Machine against this storage and tried to make it fail (shuting down glusterfs VMs) with no success
​

Result is satisfactory, but I can not afford 2 additional machines for Glusterfs...

My next test would be to add another disk to each Proxmox Server, and use it for creating a new Gluster volume, that could be shared between servers..

This seems similar approach than using DRBD (I´ve never tested DRBD) but glusterfs seems easier to implement, and has the potential advantage of using more that 2 servers.

Questions:

Can Glusterfs-server coexist with Proxmox ?
If Yes, How much would impact replication on Proxmox performance (let say 3 Proxmox Servers hosting a Glusterfs volume, replica 2, to host all(10) the VM's raw disks)
Any one having such configuration running ?
​

Regards.

 

[dietmar]

Re: Proxmox 3.1 GlusterFS Setup, On SAME Servers as Promox

Can Glusterfs-server coexist with Proxmox ?

yes

If Yes, How much would impact replication on Proxmox performance (let say 3 Proxmox Servers hosting a Glusterfs volume, replica 2, to host all(10) the VM's raw disks)

GlusterFS recovery is incredible slow (by design) for large files, because it has to read the whole data (like rsync).

 

[arifkhumaidi]

Re: Proxmox 3.1 GlusterFS Setup, On SAME Servers as Promox

hello, i'm trying to mount glusterfs, with web interface proxmox.
creating is success. but the system can't mount the content.
should i mounting manually in /etc/fstab, how to do that?
add line like this
debian01:/gv00 /mnt/pve/glusterID/ glusterfs defaults,_netdev 0 0

i got the following error :
Mount failed. Please check the log file for more details.

#dmesg | tail
EXT3-fs (dm-1): using internal journal
EXT3-fs (dm-1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting. Commit interval 5 seconds
EXT3-fs (sda2): using internal journal
EXT3-fs (sda2): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting. Commit interval 5 seconds
EXT3-fs (dm-1): using internal journal
EXT3-fs (dm-1): mounted filesystem with ordered data mode


# tail -f /var/log/glusterfs/mnt-.log

[glusterfsd.c:1002:cleanup_and_exit] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xcd) [0x7f75f3b2952d] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_handle_reply+0xa4) [0x7f75f3b291a4] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x449) [0x7f75f41e45a9]))) 0-: received signum (0), shutting down
[2014-08-21 10:33:17.558390] I [fuse-bridge.c:5260:fini] 0-fuse: Unmounting '/mnt/'.

 

[arifkhumaidi]

Re: Proxmox 3.1 GlusterFS Setup, On SAME Servers as Promox

i got it, i use gluster version 3.2
http://forum.proxmox.com/threads/16494-GlusterFS-issues