Proxmox and GlusterFS

Discussion in 'Proxmox VE: Installation and configuration' started by Meinhard, Jul 7, 2019.

  1. Meinhard

    Meinhard New Member

    Joined:
    Jul 7, 2019
    Messages:
    3
    Likes Received:
    0
    Hello,

    I have a Proxmox cluster of two older HPE servers (DL380 gen7). These have a built-in hardware RAID controller on which 8 plates each with 146 GB hang. From this I built a RAID5 with a spare.

    I installed Debian / Stretch and then Proxmox after the official instructions. Then I set up GlusterFS, also according to instructions.

    The two servers are connected via 10 GBE (DAC cable) with each other. This link is also used by the GlusterFS.

    There are about 15 VMs with Linux (also Debian / Stretch). Most of the VMs hardly make up an IO load. I installed Icinga with pnp4nagios in a VM. That makes a bit of IO load by writing the RRDs (despite rrdcached).
    The VM has crashing the file system (btrfs) several times in the last few months, so hard that I had to reinstall the VM each time. Once I was able to follow the live - the kernel complained:
    2019 Jun 3 11:30:34 lnxicinga01 BTRFS: error (device vda1) in cleanup_transaction: 1846: errno = -5 IO failure

    Apparently there are always "hangers" in GluserFS. In the simplest case, after such a problem, the file system is only mounted on ro. In the worst case GlusterFS shows me a split brain - then the file system is also heavily damaged in most cases.

    For two weeks I have run a second VM with Icinga and pnp4nagios. Since then, the mentioned problems occur at least once a day. On Friday, the file systems have completely passed away with three VMs - I have to rebuild these VMs completely again :-(

    I have no idea what the problem could be. The RAID controllers show no errors, all disks work, there are no special kernel messages on the two Proxmox servers. The network connection also shows no problems or failures.

    Is stability basically not good at GlusterFS? My problem is that I do not have my own SAN, so all data should be distributed through the Proxmox machines.

    I would be very grateful for any kind of help or suggestions.

    many Greetings
    Meinhard
     
  2. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,763
    Likes Received:
    316
    Hi,
    The GlusterFS server is not part of ProxmoxVE.
    On which version of GlusterFS server you are?
    Generall, it is recommended to upgrade to the current version of GlusterFS server.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Meinhard

    Meinhard New Member

    Joined:
    Jul 7, 2019
    Messages:
    3
    Likes Received:
    0
    Hi Werner,

    I use the versions from Debian / Stretch:
    Code:
    root@pve-c0-n0:~# dpkg -l gluster\*
    Gewünscht=Unbekannt/Installieren/R=Entfernen/P=Vollständig Löschen/Halten
    | Status=Nicht/Installiert/Config/U=Entpackt/halb konFiguriert/
             Halb installiert/Trigger erWartet/Trigger anhängig
    |/ Fehler?=(kein)/R=Neuinstallation notwendig (Status, Fehler: GROSS=schlecht)
    ||/ Name                 Version         Architektur     Beschreibung
    +++-====================-===============-===============-=============================================
    ii  glusterfs-client     3.8.8-1         amd64           clustered file-system (client package)
    ii  glusterfs-common     3.8.8-1         amd64           GlusterFS common libraries and translator mod
    ii  glusterfs-server     3.8.8-1         amd64           clustered file-system (server package)
    
    I'm just astonished that version 6.3.1 is available on gluster.org - even with packages for Debian.

    Is using the packages from gluster.org safe? Do you think that could be the reason for my problems - the much older version of the packages in Debian / Stretch?

    Thank you and best regards
    Meinhard
     
  4. Meinhard

    Meinhard New Member

    Joined:
    Jul 7, 2019
    Messages:
    3
    Likes Received:
    0
    Hello,

    I'm not sure if the cluster should just rebuild - with Debian / Buster and Proxmox 6. I would put it then with three nodes.

    However, I'm completely unsure what to do for storage replication - GlusterFS or CEPH ?!

    Any advice?

    Many Thanks
    Meinhard

    PS: I would then set up, initialize and run GlusterFS with the debian packages from gluster.org.
     
  5. ricardoj

    ricardoj Member

    Joined:
    Oct 16, 2018
    Messages:
    67
    Likes Received:
    2
    Hi,

    It's good to know that someone else is testing GlusterFS.

    I put a message some days ago ( here ) asking for comments about GlusterFS and its performance.

    So far my tests indicates its an option for a shared storage but more tests must be done.

    I'm running the most uptodate GlusterFS version ( 6.3.1 ) as weel as BtrFS ( 4.20 ) without any fail or disk / data corruption.

    I have BtrFS working on a NAS under heavy traffic for months.

    You can see another thread about BtrFS here.

    I always use RAID 1 for BtrFS without any disk array on top of the disks ( JBOD ) in a commodity hardware.

    My main reason for testing BtrFS is ZFS performance on low end hardware.On high end hardware ZFS works very well.

    Now back to GlusterFS I'm also testing it with "shard" on a Gigabit Ethernet with jumbo frame and the traffic and time to sync are low.

    Regards,

    Ricardo Jorge
     
  6. ricardoj

    ricardoj Member

    Joined:
    Oct 16, 2018
    Messages:
    67
    Likes Received:
    2
    Hi,

    For GlusterFS Read Only issue, please, see this.

    Regards,

    Ricardo Jorge
     
  7. Erk

    Erk Member

    Joined:
    Dec 11, 2009
    Messages:
    149
    Likes Received:
    3
    I set up a glusterfs test on two identical Proxmox servers to try and learn a bit about it.
    On both servers I created a 200GB xfs formatted partition /dev/sda4 mounted on /storage and created/started the glusterfs volume also called storage.

    I added the storage volume using the Proxmox GUI which I presume just front ends the glusterfs client in this case. It mounted the storage on /mnt/pve/storage. I then uploaded a 1.8GB .iso file to the storage using the Proxmox content upload section of the GUI. All good and working but I noticed something strange hence this question.

    Here is the output of the df -h command

    # df -h
    Filesystem Size Used Avail Use% Mounted on
    udev 3.9G 0 3.9G 0% /dev
    tmpfs 789M 9.0M 780M 2% /run
    /dev/mapper/pve-root 32G 1.8G 29G 6% /
    tmpfs 3.9G 43M 3.9G 2% /dev/shm
    tmpfs 5.0M 0 5.0M 0% /run/lock
    tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
    /dev/sda4 200G 2.1G 198G 2% /storage
    /dev/fuse 30M 16K 30M 1% /etc/pve
    tmpfs 789M 0 789M 0% /run/user/1000
    bne1:storage 200G 4.1G 196G 3% /mnt/pve/storage


    Notice that /mnt/pve/storage shows 4.1GB used when /storage only shows 2.1GB used.
    Where has 2GB gone?

    When I do a du -sh in /mnt/pve/storage the sizes are correct.

    :/mnt/pve/storage# du -sh *
    4.0K dump
    4.0K images
    1.8G template


    Is the discrepancy just some block rounding error or will it get worse?
     
  8. ricardoj

    ricardoj Member

    Joined:
    Oct 16, 2018
    Messages:
    67
    Likes Received:
    2
    Hi,

    Try this : gluster volume status <volume-name> detail

    and see what happens.

    Regards,

    Ricardo Jorge
     
  9. Erk

    Erk Member

    Joined:
    Dec 11, 2009
    Messages:
    149
    Likes Received:
    3

    Hi Ricardo,
    that seems to show the correct size.

    root@bne1:/mnt/pve/storage# gluster volume status storage detail
    Status of volume: storage
    ------------------------------------------------------------------------------
    Brick : Brick bne1:/storage
    TCP Port : 49152
    RDMA Port : 0
    Online : Y
    Pid : 1830
    File System : xfs
    Device : /dev/sda4
    Mount Options : rw,relatime,attr2,inode64,noquota
    Inode Size : N/A
    Disk Space Free : 197.9GB
    Total Disk Space : 199.9GB
    Inode Count : 104857600
    Free Inodes : 104857560
    ------------------------------------------------------------------------------
    Brick : Brick bne2:/storage
    TCP Port : 49153
    RDMA Port : 0
    Online : Y
    Pid : 1684
    File System : xfs
    Device : /dev/sda4
    Mount Options : rw,relatime,attr2,inode64,noquota
    Inode Size : N/A
    Disk Space Free : 197.9GB
    Total Disk Space : 199.9GB
    Inode Count : 104857600
    Free Inodes : 104857560



    Must just be a glitch in the df -h command showing the incorrect size.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice