Proxmox and GlusterFS

[Meinhard]

Hello,

I have a Proxmox cluster of two older HPE servers (DL380 gen7). These have a built-in hardware RAID controller on which 8 plates each with 146 GB hang. From this I built a RAID5 with a spare.

I installed Debian / Stretch and then Proxmox after the official instructions. Then I set up GlusterFS, also according to instructions.

The two servers are connected via 10 GBE (DAC cable) with each other. This link is also used by the GlusterFS.

There are about 15 VMs with Linux (also Debian / Stretch). Most of the VMs hardly make up an IO load. I installed Icinga with pnp4nagios in a VM. That makes a bit of IO load by writing the RRDs (despite rrdcached).
The VM has crashing the file system (btrfs) several times in the last few months, so hard that I had to reinstall the VM each time. Once I was able to follow the live - the kernel complained:
2019 Jun 3 11:30:34 lnxicinga01 BTRFS: error (device vda1) in cleanup_transaction: 1846: errno = -5 IO failure

Apparently there are always "hangers" in GluserFS. In the simplest case, after such a problem, the file system is only mounted on ro. In the worst case GlusterFS shows me a split brain - then the file system is also heavily damaged in most cases.

For two weeks I have run a second VM with Icinga and pnp4nagios. Since then, the mentioned problems occur at least once a day. On Friday, the file systems have completely passed away with three VMs - I have to rebuild these VMs completely again :-(

I have no idea what the problem could be. The RAID controllers show no errors, all disks work, there are no special kernel messages on the two Proxmox servers. The network connection also shows no problems or failures.

Is stability basically not good at GlusterFS? My problem is that I do not have my own SAN, so all data should be distributed through the Proxmox machines.

I would be very grateful for any kind of help or suggestions.

many Greetings
Meinhard

 

[wolfgang]

Hi,

Is stability basically not good at GlusterFS?

The GlusterFS server is not part of ProxmoxVE.
On which version of GlusterFS server you are?
Generall, it is recommended to upgrade to the current version of GlusterFS server.

 

[Meinhard]

Hi Werner,

I use the versions from Debian / Stretch:

root@pve-c0-n0:~# dpkg -l gluster\*
Gewünscht=Unbekannt/Installieren/R=Entfernen/P=Vollständig Löschen/Halten
| Status=Nicht/Installiert/Config/U=Entpackt/halb konFiguriert/
         Halb installiert/Trigger erWartet/Trigger anhängig
|/ Fehler?=(kein)/R=Neuinstallation notwendig (Status, Fehler: GROSS=schlecht)
||/ Name                 Version         Architektur     Beschreibung
+++-====================-===============-===============-=============================================
ii  glusterfs-client     3.8.8-1         amd64           clustered file-system (client package)
ii  glusterfs-common     3.8.8-1         amd64           GlusterFS common libraries and translator mod
ii  glusterfs-server     3.8.8-1         amd64           clustered file-system (server package)

I'm just astonished that version 6.3.1 is available on gluster.org - even with packages for Debian.

Is using the packages from gluster.org safe? Do you think that could be the reason for my problems - the much older version of the packages in Debian / Stretch?

Thank you and best regards
Meinhard

 

[Meinhard]

Hello,

I'm not sure if the cluster should just rebuild - with Debian / Buster and Proxmox 6. I would put it then with three nodes.

However, I'm completely unsure what to do for storage replication - GlusterFS or CEPH ?!

Any advice?

Many Thanks
Meinhard

PS: I would then set up, initialize and run GlusterFS with the debian packages from gluster.org.

 

[ricardoj]

Hi,

It's good to know that someone else is testing GlusterFS.

I put a message some days ago ( here ) asking for comments about GlusterFS and its performance.

So far my tests indicates its an option for a shared storage but more tests must be done.

I'm running the most uptodate GlusterFS version ( 6.3.1 ) as weel as BtrFS ( 4.20 ) without any fail or disk / data corruption.

I have BtrFS working on a NAS under heavy traffic for months.

You can see another thread about BtrFS here.

I always use RAID 1 for BtrFS without any disk array on top of the disks ( JBOD ) in a commodity hardware.

My main reason for testing BtrFS is ZFS performance on low end hardware.On high end hardware ZFS works very well.

Now back to GlusterFS I'm also testing it with "shard" on a Gigabit Ethernet with jumbo frame and the traffic and time to sync are low.

Regards,

Ricardo Jorge

 

[ricardoj]

Hi,

For GlusterFS Read Only issue, please, see this.

Regards,

Ricardo Jorge

 

[Erk]

I set up a glusterfs test on two identical Proxmox servers to try and learn a bit about it.
On both servers I created a 200GB xfs formatted partition /dev/sda4 mounted on /storage and created/started the glusterfs volume also called storage.

I added the storage volume using the Proxmox GUI which I presume just front ends the glusterfs client in this case. It mounted the storage on /mnt/pve/storage. I then uploaded a 1.8GB .iso file to the storage using the Proxmox content upload section of the GUI. All good and working but I noticed something strange hence this question.

Here is the output of the df -h command

# df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 789M 9.0M 780M 2% /run
/dev/mapper/pve-root 32G 1.8G 29G 6% /
tmpfs 3.9G 43M 3.9G 2% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda4 200G 2.1G 198G 2% /storage
/dev/fuse 30M 16K 30M 1% /etc/pve
tmpfs 789M 0 789M 0% /run/user/1000
bne1:storage 200G 4.1G 196G 3% /mnt/pve/storage

Notice that /mnt/pve/storage shows 4.1GB used when /storage only shows 2.1GB used.
Where has 2GB gone?

When I do a du -sh in /mnt/pve/storage the sizes are correct.

:/mnt/pve/storage# du -sh *
4.0K dump
4.0K images
1.8G template

Is the discrepancy just some block rounding error or will it get worse?

 

[ricardoj]

Hi,

Try this : gluster volume status <volume-name> detail

and see what happens.

Regards,

Ricardo Jorge

 

[Erk]

Hi,

Try this : gluster volume status <volume-name> detail

and see what happens.

Regards,

Ricardo Jorge

Hi Ricardo,
that seems to show the correct size.

root@bne1:/mnt/pve/storage# gluster volume status storage detail
Status of volume: storage
------------------------------------------------------------------------------
Brick : Brick bne1:/storage
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 1830
File System : xfs
Device : /dev/sda4
Mount Options : rw,relatime,attr2,inode64,noquota
Inode Size : N/A
Disk Space Free : 197.9GB
Total Disk Space : 199.9GB
Inode Count : 104857600
Free Inodes : 104857560
------------------------------------------------------------------------------
Brick : Brick bne2:/storage
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 1684
File System : xfs
Device : /dev/sda4
Mount Options : rw,relatime,attr2,inode64,noquota
Inode Size : N/A
Disk Space Free : 197.9GB
Total Disk Space : 199.9GB
Inode Count : 104857600
Free Inodes : 104857560


Must just be a glitch in the df -h command showing the incorrect size.